fix(listen): recover when the CLI session expires server-side - #321
Closed
leggetter wants to merge 1 commit into
Closed
fix(listen): recover when the CLI session expires server-side#321leggetter wants to merge 1 commit into
leggetter wants to merge 1 commit into
Conversation
The websocket proxy stores CLI sessions in Redis with a TTL. When the session key is missing, the server expects the CLI to send session- recreation headers on connect, and otherwise closes the connection with code 4001 (SESSION_EXPIRED). The CLI sent neither the headers nor handled the close code: it created its session once per listen run and reconnected with the same dead session ID forever, flapping between "Connected" and "Connection lost" while every event failed with CLI_UNAVAILABLE. Two complementary fixes: - Send X-Webhook-Ids (comma-separated connection IDs) and X-Session-Filters (base64-encoded JSON) on every websocket connect so the server can transparently recreate an expired session under the same session ID. - Detect a session-expired signal (close code 4001 after connecting, or the legacy "Unknown WebSocket ID." upgrade rejection) and create a fresh session via the API before the next reconnect attempt instead of retrying the dead session ID. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MH9LQENoLJSawdD7X4yy2h
Collaborator
Author
|
Closing in favor of #322, which covers the same core fix (session-recreation headers Two things from this branch worth carrying over to #322 during review:
Related server-side fix (protects already-released CLIs when a session vanishes from Redis mid-connection): hookdeck/core#5402. Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
The platform's websocket proxy (
hookdeck/core#4477) stores CLI sessions in Redis with a TTL instead of PostgreSQL. When a session key is missing server-side, the proxy expects the CLI to send session-recreation headers on connect (x-webhook-ids,x-session-filters) — a contract described inserver/workers/websocketProxy.tsas something "future CLI versions" will implement — and otherwise closes the connection with code4001 SESSION_EXPIRED.No released CLI version sends those headers or handles that close code. The CLI creates its session once per
listenrun and reuses the same session ID on every reconnect, so once the server-side session disappears (Redis TTL expiry, eviction/failover, or a miss in the PG→Redis migration bridge) the CLI is permanently wedged: it flaps between "Connected" and "Connection lost, reconnecting..." with a dead session ID while every event fails withCLI_UNAVAILABLE. Related to the connectivity reports investigated in the internal Slack thread from 2026-08-03 (community report ofCLI_UNAVAILABLEwith a running CLI).Changes
Two complementary fixes:
Session-recreation headers on every connect — the websocket client now sends
X-Webhook-Ids(comma-separated connection IDs) andX-Session-Filters(base64-encoded JSON, matching the server'sparseSessionRecreationHeaders) with the upgrade request. When the server can't find the session, it transparently recreates it under the same session ID instead of rejecting the connection. This is the primary fix: with these headers, the expired-session case self-heals with no visible interruption.Session-expired recovery in the reconnect loop — the client records a session-expired signal when the server closes with code
4001after connecting (SESSION_EXPIRED/INVALID_SESSION_DATA) or rejects the upgrade with the legacyUnknown WebSocket ID.error. On the next reconnect attempt, the proxy creates a fresh session via the API instead of retrying the dead session ID. This is defense-in-depth for cases the headers can't fix (e.g. malformed recreation data).pkg/listen/proxy/proxy.goalso picked up a one-line pre-existinggofmtalignment fix in theConfigstruct.Testing
pkg/websocket/client_test.go:X-Webhook-Ids/X-Session-Filters(and base64 round-trips) when configured, and omit them otherwise;4001close from a real (httptest) websocket server setsSessionExpired();1001close does not.go build ./...,go vet, andgo test ./...pass locally, except the pre-existingpkg/listen/healthcheckTestCheckServerHealth_DefaultPorts/HTTPS_default_portfailure, which also fails on a clean checkout in this environment (cannot bind port 443) and is unrelated.hookdeck listen, delete thecses_*hash from Redis, and confirm the CLI recovers (headers path) and that a forced4001close triggers session recreation (recovery path).🤖 Generated with Claude Code
https://claude.ai/code/session_01MH9LQENoLJSawdD7X4yy2h
Generated by Claude Code