Back to Signal Feed
CodeTracked since May 19, 2026

mirrord disables HTTP/2 pooling to eliminate stale-channel failures

This change fixes a production-impacting retry failure in mirrord by disabling HTTP/2 connection pooling in `ClientStore`; reusing pooled `SendRequest` objects after an idle GOAWAY left a closed sender in the pool, so subsequent retries repeatedly failed with `channel closed`, and forcing a new connection per request removes that failure loop.

mirrordClientStoreHTTP/2gRPC

What Happened

  • This change fixes a production-impacting retry failure in mirrord by disabling HTTP/2 connection pooling in `ClientStore`; reusing pooled `SendRequest` objects after an idle GOAWAY left a closed sender in the pool, so subsequent retries repeatedly failed with `channel closed`, and forcing a new connection per request removes that failure loop.
  • This change fixes a production-impacting retry failure in mirrord by disabling HTTP/2 connection pooling in `ClientStore`; reusing pooled `SendRequest` objects after an idle GOAWAY left a closed sender in the pool, so subsequent retries repeatedly failed with `channel closed`, and forcing a new connection per request removes that failure loop.
  • 1 evidence item attached for review.

What is Different

Before

Scattered source updates, isolated context, and manual follow-up across multiple feeds.

Now

The PR changes the mirrord HTTP/2 client path so that pooling is effectively turned off (`should_enable_connection_pooling() -> false`), ensuring each request uses a newly created connection instead of reusing possibly closed pooled senders, which restores retry effectiveness.

Why Track This

Why It Matters

Operators and developers running mirrord in gRPC/HTTP/2 environments with short idle timeouts (for example Quarkus via Vert.x in Kubernetes meshes) should see fewer intermittent `channel closed` failures and more successful retries, reducing flaky behavior during local traffic-steering and integration testing. The update works by preventing `get_with_pooling()` from returning stale `http2::SendRequest` instances after GOAWAY-triggered closure, so each request starts with a fresh channel. Watch for connection-establishment cost under high request volume and whether a future targeted fix (e.g., discarding only closed senders) is needed to regain pooling benefits without reintroducing stale-channel reuse bugs.

Impact

Operators and developers running mirrord in gRPC/HTTP/2 environments with short idle timeouts (for example Quarkus via Vert.x in Kubernetes meshes) should see fewer intermittent `channel closed` failures and more successful retries, reducing flaky behavior during local traffic-steering and integration testing. The update works by preventing `get_with_pooling()` from returning stale `http2::SendRequest` instances after GOAWAY-triggered closure, so each request starts with a fresh channel. Watch for connection-establishment cost under high request volume and whether a future targeted fix (e.g., discarding only closed senders) is needed to regain pooling benefits without reintroducing stale-channel reuse bugs.

What To Watch Next

  • Watch whether mirrord becomes a repeated pattern.
  • Track follow-up changes around LLMOps.
  • Compare future signals against this evidence trail.
  • Re-check risk flags: increased_connection_setup_overhead, high_request_rate_latency_regression.
Open Topic TimelineOpen Technical EventOpen Original Sourceincreased_connection_setup_overhead / high_request_rate_latency_regression / loss_of_connection_reuse_benefit / need_for_lifecycle_notification_check

Supporting Evidence

GITHUB PULL REQUESTHigh Trust

metalbear-co/mirrord PR #4274: fix: disable HTTP/2 connection pooling to prevent "channel closed" errors

The PR sets `should_enable_connection_pooling()` to `false`, verified by repeated test runs where `channel closed` errors disappear when pooling is disabled.