Skip to content

fix(acp): coordinate HTTP rate-limit retries across RestClient clones - #5620

Open
dbett4 wants to merge 1 commit into
block:mainfrom
dbett4:acp-http-rate-limit-gate
Open

fix(acp): coordinate HTTP rate-limit retries across RestClient clones#5620
dbett4 wants to merge 1 commit into
block:mainfrom
dbett4:acp-http-rate-limit-gate

Conversation

@dbett4

@dbett4 dbett4 commented Aug 12, 2026

Copy link
Copy Markdown

Problem

RestClient::request_with_retry treats HTTP 429 as a generic transient error — is_retriable_status groups it with 502/503/504 — so a rate-limited request retries on the fixed 500ms/1s/2s ladder and gives up after ~3.5s total. Two consequences:

  • Retry-After is never read. A relay that answers Retry-After: 30 sees four attempts land inside its own limit window, and the call still fails.
  • RestClient is Clone, holds no rate-limit state, and is Arc-shared with spawned prompt tasks by design. Concurrent callers each run the ladder independently, so one caller's 429 does not slow any other — a limit produces a retry storm instead of a drain.

The WebSocket path in this same file already handles this properly: BgState::rate_limit_gate and rate_limited_pending park admission-counted frames until the gate clears. This change gives the HTTP bridge path equivalent behavior.

Change

  • Process-local HTTP_RATE_GATE shared by every RestClient clone. A 429 arms or extends it; a shorter hint never shortens an already-armed gate.
  • Retry-After parsed as integer seconds, floored to 5s when absent/unparseable/< 2s and capped at 300s, with the existing jitter applied.
  • A 429 no longer consumes the plain backoff sleep — the attempt waits on the gate instead.
  • HTTP_BRIDGE_PERMITS serializes bridge admission so a request already queued re-checks the gate after a peer arms it, rather than racing past it.

Tests

  • parse_http_retry_after_secs_valid_and_invalid — integer seconds honored; HTTP-date and malformed values fall through to the conservative default.
  • http_rate_gate_extends_without_shortening and http_rate_gate_missing_retry_after_uses_default — gate arithmetic.
  • concurrent_rest_client_clones_share_http_rate_gate — wiremock server, four concurrent clones, first response 429 + Retry-After: 2. Asserts all four calls succeed, total upstream hits stay within 2..=6, and — the assertion that actually does the work — that the second upstream hit lands at least 1500ms after the first, i.e. the queued clone waited out the hint rather than racing past it.

Verified as a detector, not just a passing test: with the gate wait, the permit acquisition, and the 429 arm removed from request_with_retry (helpers and tests left intact, so the only delta is the fix itself), this test fails on the first queued clone bypassed the shared Retry-After gate. It is not green-by-default.

cargo test -p buzz-acp on this branch: 740 passed, 0 failed (plus 9 in pool_lifecycle_state).

Notes for review

  • Adds wiremock as a dev-dependency; the Cargo.lock delta is entirely that.
  • HTTP_BRIDGE_PERMITS is Semaphore::const_new(1), so bridge requests serialize process-wide even when no gate is armed. That is the deliberate trade-off here — it is what lets a queued request observe a gate armed by a peer instead of racing past it — but it is a real throughput cost on the happy path and the part I would most expect pushback on. The alternative is a wider permit count with a re-check after acquisition, accepting a small stampede window on the first 429. Happy to switch if you prefer that shape.
  • Supersedes fix(acp): coordinate HTTP rate-limit retries #3618, which bundled this fix with five unrelated agent-credential commits under a title describing only this one. Closing that in favor of this.

Signed-off-by: Dave Bettner <dbett4@gmail.com>
@dbett4
dbett4 requested a review from a team as a code owner August 12, 2026 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant