Pooled connection is reused after the daemon closed it, so any POST can fail with NoHttpResponseException (testcontainers/podman) #2682

Summary

ApacheDockerHttpClientImpl never expires or revalidates an idle pooled connection, so when a daemon (like podman) closes one on its own idle timer, docker-java hands the dead connection to the next request.
An idempotent request survives (HttpClient5 retries it); a POST — /containers/create, /containers/{id}/exec, /networks/create — fails with:

java.lang.RuntimeException: org.apache.hc.core5.http.NoHttpResponseException: localhost:2375 failed to respond

This is the bug behind testcontainers-java#7310 and testcontainers-java#7593, both open since 2023 and both still reproducing. It has never had a correctly scoped issue here — #2322 is the same error message, but it was read as a URL-parsing problem and closed as stale.

localhost:2375 is a red herring. For every socket transport, ApacheDockerHttpClientImpl:78 synthesises a placeholder authority:

case "unix":
case "npipe":
    pathPrefix = "";
    host = new HttpHost(dockerHost.getScheme(), "localhost", 2375);

No TCP connection to port 2375 is ever attempted. Reports of this bug are routinely misfiled as "docker-java ignores my socket path" because of it.

Reproduction

No testcontainers, no test framework — plain docker-java against DOCKER_HOST. Warm up a connection, idle, then POST.

DockerClientConfig config = DefaultDockerClientConfig.createDefaultConfigBuilder().build();

for (int idleSeconds : new int[] {2, 4, 6, 8, 10, 15, 30}) {
    try (DockerHttpClient http = new ZerodepDockerHttpClient.Builder()
             .dockerHost(config.getDockerHost()).build();
         DockerClient client = DockerClientImpl.getInstance(config, http)) {
        client.pingCmd().exec();                 // opens a connection, returns it to the pool
        Thread.sleep(idleSeconds * 1000L);       // the daemon's idle timer runs out in here
        client.createContainerCmd(IMAGE).exec(); // POST on the pooled connection
    }
}

Against podman 5.8.6 over npipe, docker-java 3.7.1, identical across three runs:

idle(s) | outcome
--------+-----------------------------------------------------------
      7 | OK
      8 | OK
      9 | OK
     10 | NoHttpResponseException: localhost:2375 failed to respond
     15 | NoHttpResponseException: localhost:2375 failed to respond
     30 | NoHttpResponseException: localhost:2375 failed to respond

A hard cliff at 10s. Every failure is followed by an immediate retry on a fresh connection that succeeds, so the daemon is up throughout — only the pooled connection was dead.

Why

Three things have to line up, and all three are in ApacheDockerHttpClientImpl.

1. The daemon closes idle connections and does not say so. Podman's API server sets Go's http.Server.IdleTimeout to twice its service_timeout (pkg/api/server/server.go:82, IdleTimeout: opts.Timeout * 2; pkg/api/server/idle/tracker.go documents the 2x relationship).
service_timeout defaults to 5s, giving the 10s window measured above.

Measured a second way, without docker-java in the path at all: a raw socket to the pipe, one completed GET /_ping, then waiting to be hung up on, gives 9843–9976 ms across five runs. Anything gating a test on this should probe that behaviour rather than the daemon's identity — a podman with service_timeout=0 keeps connections open and cannot exhibit the bug either.

It advertises nothing. Verified on podman 5.8.6 — no Keep-Alive, no Connection header on /_ping or /version, checked both through docker-java and with curl -D - directly against /run/podman/podman.sock inside the machine, so it is not a relay stripping headers. Adding a Keep-Alive header was proposed in podman-container-tools/podman#17640 and declined: podman uses Go's stdlib HTTP server, which does not emit one. That issue was closed as "mostly a client issue".

2. Nothing bounds the idle life of a pooled connection. There is no setConnectionKeepAlive(...) on the default RequestConfig, so DefaultConnectionKeepAliveStrategy — finding no Keep-Alive header — falls back to RequestConfig's default of 3 minutes. The client believes a connection closed seconds ago is good for minutes.

3. Nothing catches it on the way out, and POST is not retried.
ApacheDockerHttpClientImpl:126 disables revalidation on lease:

.setValidateAfterInactivity(TimeValue.NEG_ONE_SECOND)

and no HttpRequestRetryStrategy is installed, so DefaultHttpRequestRetryStrategy applies: it retries NoHttpResponseException for idempotent methods only.

That last point explains why this reads as random flakiness rather than a consistent failure. A GET on a dead connection is retried and recovers silently — HttpClient5 even logs it:

HttpRequestRetryExec - recoverable I/O exception (NoHttpResponseException) caught when sending
request to npipe://localhost:2375; request will be automatically re-executed

So whether a caller sees anything depends on which method happens to land on the dead connection first. In testcontainers, GenericContainer.start() usually survives, because its first call after an idle gap is an image inspect (a GET) that absorbs the failure and replaces the connection — while execInContainer issues POST /containers/{id}/exec first and has nothing to hide behind.
That is exactly the split between testcontainers-java#7593 (exec, fails) and the many reports where container startup is intermittent.

Why the obvious fix is blocked

Re-enabling revalidation looks like the one-line answer. It is not, and the reason is worth recording, because every previous report of this bug has missed it.

setValidateAfterInactivity(NEG_ONE_SECOND) was set deliberately, by c61da29 ("Fix #1726 by disabling stale connection checking in AHC5"). #1726 was a hang: the validating read blocked forever on a domain socket, because ApacheDockerHttpClientImpl:120 sets setSoTimeout(ZERO_MILLISECONDS) so that long-lived attach and log streams are not cut off.

httpcore5 does bound the probe — BHttpConnectionBase.isStale() calls fillInputBuffer with a 1ms STALE_CHECK_TIMEOUT, which does socket.setSoTimeout(1) around the read. But NamedPipeSocket and UnixSocket never implement setSoTimeout. Inherited from java.net.Socket, the call creates an unrelated SocketImpl to hold the value and has no effect on reads from the pipe or channel, so the 1ms bound is silently a no-op.

Measured: re-enabling revalidation on podman over npipe does fix the failure, and costs this.

idle(s) | round trip with revalidation |  without
--------+------------------------------+----------------------------
      2 |                    19285 ms  |  ~400 ms
      6 |                    16102 ms  |  ~400 ms
     10 |                    10311 ms  |  NoHttpResponseException
     15 |                    10317 ms  |  NoHttpResponseException
     30 |                    10316 ms  |  NoHttpResponseException

Every probe on a healthy connection blocks for the daemon's entire idle window before returning. The 10.3s floor is podman's 10s. That is #1726, in numbers.

Implementing setSoTimeout on NamedPipeSocket restored the latency (127–420ms) but introduced something worse: POST /containers/create began receiving the previous response's body. With a 2-byte /_ping warm-up, CreateContainerResponse was handed the literal OK. The cause is structural rather than a coding slip — ApacheResponse.close() calls request.abort() without draining, then the connection returns to the pool on the assumption that the socket stream sits on a message boundary. Any buffer added beneath httpcore5 is state it cannot see or reset, and a
bounded read on an async channel needs either read-ahead or cancellation. Silent desynchronisation of the Docker API stream is a worse outcome than the original bug, so this line was abandoned.

A retry strategy is the other candidate, and it is not unambiguously safe either. NoHttpResponseException says no response arrived, not that nothing was sent, so retrying a POST on it is more permissive than Go's client — which retries a non-idempotent request only when nothing
was written (net/http/transport.go:846-890).

Environment

• docker-java 3.7.1 (also reproduced against main)
• podman 5.8.6, podman machine on WSL, DOCKER_HOST=npipe:////./pipe/podman-machine-default
• Windows 11, JDK 21
• Also reported on macOS and Linux (unix://) in testcontainers-java#7310

Related

• testcontainers-java#7310 — open, correct diagnosis, no fix; the last "still a problem" is 2026-06
• testcontainers-java#7593 — open, execInContainer, same message
• testcontainers-java#6667 — added a retry to network creation only, shipped in 1.18.0; this is why
  some reporters say an upgrade fixed it while others still see it
• /v1.32/networks/create "failed to respond" podman-container-tools/podman#17640 — closed as a client-side issue; the Keep-Alive header was declined
• Both Apache and ZeroDep clients ignore socket URLs #2322 — same error message, misread as URL parsing, closed stale
• DomainSocket.read(Native Method) never timeout / hang for Docker Stop Container operation #1726 / c61da29 — why revalidation is currently disabled

Fix

With Claude Code help, I made a 1-line fix, added unitary and integration tests. Both are failing without the fix and are passed with the fix.
The PR will soon be provided.