Skip to content

fix(listen): recover when the CLI session expires server-side - #321

Closed
leggetter wants to merge 1 commit into
mainfrom
claude/hookdeck-cli-connectivity-uin9pd
Closed

fix(listen): recover when the CLI session expires server-side#321
leggetter wants to merge 1 commit into
mainfrom
claude/hookdeck-cli-connectivity-uin9pd

Conversation

@leggetter

Copy link
Copy Markdown
Collaborator

Context

The platform's websocket proxy (hookdeck/core #4477) stores CLI sessions in Redis with a TTL instead of PostgreSQL. When a session key is missing server-side, the proxy expects the CLI to send session-recreation headers on connect (x-webhook-ids, x-session-filters) — a contract described in server/workers/websocketProxy.ts as something "future CLI versions" will implement — and otherwise closes the connection with code 4001 SESSION_EXPIRED.

No released CLI version sends those headers or handles that close code. The CLI creates its session once per listen run and reuses the same session ID on every reconnect, so once the server-side session disappears (Redis TTL expiry, eviction/failover, or a miss in the PG→Redis migration bridge) the CLI is permanently wedged: it flaps between "Connected" and "Connection lost, reconnecting..." with a dead session ID while every event fails with CLI_UNAVAILABLE. Related to the connectivity reports investigated in the internal Slack thread from 2026-08-03 (community report of CLI_UNAVAILABLE with a running CLI).

Changes

Two complementary fixes:

  1. Session-recreation headers on every connect — the websocket client now sends X-Webhook-Ids (comma-separated connection IDs) and X-Session-Filters (base64-encoded JSON, matching the server's parseSessionRecreationHeaders) with the upgrade request. When the server can't find the session, it transparently recreates it under the same session ID instead of rejecting the connection. This is the primary fix: with these headers, the expired-session case self-heals with no visible interruption.

  2. Session-expired recovery in the reconnect loop — the client records a session-expired signal when the server closes with code 4001 after connecting (SESSION_EXPIRED / INVALID_SESSION_DATA) or rejects the upgrade with the legacy Unknown WebSocket ID. error. On the next reconnect attempt, the proxy creates a fresh session via the API instead of retrying the dead session ID. This is defense-in-depth for cases the headers can't fix (e.g. malformed recreation data).

pkg/listen/proxy/proxy.go also picked up a one-line pre-existing gofmt alignment fix in the Config struct.

Testing

  • New unit tests in pkg/websocket/client_test.go:
    • headers include X-Webhook-Ids / X-Session-Filters (and base64 round-trips) when configured, and omit them otherwise;
    • a 4001 close from a real (httptest) websocket server sets SessionExpired();
    • a normal 1001 close does not.
  • go build ./..., go vet, and go test ./... pass locally, except the pre-existing pkg/listen/healthcheck TestCheckServerHealth_DefaultPorts/HTTPS_default_port failure, which also fails on a clean checkout in this environment (cannot bind port 443) and is unrelated.
  • Not yet verified end-to-end against a server running the new websocket proxy — hence the draft. Suggested verification: start hookdeck listen, delete the cses_* hash from Redis, and confirm the CLI recovers (headers path) and that a forced 4001 close triggers session recreation (recovery path).

🤖 Generated with Claude Code

https://claude.ai/code/session_01MH9LQENoLJSawdD7X4yy2h


Generated by Claude Code

The websocket proxy stores CLI sessions in Redis with a TTL. When the
session key is missing, the server expects the CLI to send session-
recreation headers on connect, and otherwise closes the connection with
code 4001 (SESSION_EXPIRED). The CLI sent neither the headers nor
handled the close code: it created its session once per listen run and
reconnected with the same dead session ID forever, flapping between
"Connected" and "Connection lost" while every event failed with
CLI_UNAVAILABLE.

Two complementary fixes:

- Send X-Webhook-Ids (comma-separated connection IDs) and
  X-Session-Filters (base64-encoded JSON) on every websocket connect so
  the server can transparently recreate an expired session under the
  same session ID.
- Detect a session-expired signal (close code 4001 after connecting, or
  the legacy "Unknown WebSocket ID." upgrade rejection) and create a
  fresh session via the API before the next reconnect attempt instead
  of retrying the dead session ID.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MH9LQENoLJSawdD7X4yy2h
@leggetter

Copy link
Copy Markdown
Collaborator Author

Closing in favor of #322, which covers the same core fix (session-recreation headers X-Webhook-Ids / X-Session-Filters on every connect, identical wire format) plus clean close 1000 on shutdown, quiet reconnects on server closes 1001/4001, backoff reset, and race-safety — a superset of this PR.

Two things from this branch worth carrying over to #322 during review:

  1. API-based session recreation fallback (the one behavior here that Improve listen reliability with the new websocket proxy (session recreation, clean close, quiet reconnects) #322 doesn't have): on a session-expired signal, this branch created a fresh session via POST /cli-sessions before reconnecting, instead of relying solely on the server recreating the session from headers. With the server's getOrCreateSession behavior, headers-only is sufficient for SESSION_EXPIRED; the residual case is INVALID_SESSION_DATA (malformed recreation headers). Probably fine to drop — noting it so it's a conscious decision.
  2. Improve listen reliability with the new websocket proxy (session recreation, clean close, quiet reconnects) #322 needs a rebase before merge: its branch is cut from January's main (v1.6-era, pre-v2.0.0 Go SDK removal) — proxy.go there still imports hookdeck-go-sdk (hookdecksdk.Connection / connection.Id), which no longer exists on main, and GitHub already reports the PR as conflicted. client.go should rebase cleanly.

Related server-side fix (protects already-released CLIs when a session vanishes from Redis mid-connection): hookdeck/core#5402.


Generated by Claude Code

@leggetter leggetter closed this Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants