Improve listen reliability with the new websocket proxy (session recreation, clean close, quiet reconnects)

[TroyCoombs]

Summary

Improves hookdeck listen stability and UX against the new server-side websocket proxy (Core #4477), which moved CLI sessions to Redis with a TTL:

• Session recreation on reconnect: send X-Webhook-Ids (comma-separated connection IDs) and X-Session-Filters (base64-encoded JSON) headers on every connect/reconnect, so the server can recreate an expired session instead of rejecting the connection with 4001 SESSION_EXPIRED. This fixes the permanent "connected-but-broken" flap loop where a CLI would retry a dead session ID forever while all events failed with CLI_UNAVAILABLE. Filters are base64-encoded because raw UTF-8 header bytes are decoded as latin-1 by Node and would silently corrupt non-ASCII values.
• Clean close on shutdown: Ctrl+C / quit now sends a WebSocket close 1000 before teardown, letting the server tombstone the session immediately instead of holding it through the reconnect grace window (prevents events routing to a CLI that's gone).
• Quiet reconnects on intentional server closes: close codes 1001 (pod restart during a deploy) and 4001 (session expired, recreated on reconnect) now log at debug level instead of printing an alarming ERROR + "share the output with the Hookdeck team" message on every deploy. This also fixes a latent bug where ws.IsCloseError(err) was called with no codes (always false), which made the error-level close branches unreachable — genuinely unexpected close codes now surface properly.
• Backoff counter reset: the reconnect attempt counter resets after a successful connection, so backoff reflects consecutive failures rather than lifetime reconnects (long-running CLIs no longer drift toward the 10s max delay after routine deploys).
• Race-safety: websocket client connection state and the proxy's client reference are now guarded for cross-goroutine access (signal handler vs. reconnect loop), verified under go test -race.

Test plan

• go build ./..., go vet, gofmt clean
• go test -race ./pkg/websocket/ ./pkg/listen/... passes, including new tests for: session-recreation headers (content, base64 ASCII-safety, omission when unset), clean close 1000 on Stop, Stop idempotency, Stop/connect race, and quiet reconnect on server closes 1001/4001
• Manual: run hookdeck listen against prod, verify session recreation after Redis expiry and quiet reconnect through a ws-proxy deploy

Made with Cursor

[leggetter]

Pushed a merge commit (d9b6638) bringing this branch up to date with main — it was cut from January's main (v1.6 era, before the v2.0.0 Go SDK removal), so the PR had conflicts and wouldn't merge.

Resolution notes:

• pkg/listen/proxy/proxy.go: kept current main's base (pkg/hookdeck types instead of the removed hookdeck-go-sdk, shared APIClient, NoHealthcheck/Insecure health-check handling) and re-applied this branch's changes on top. One type fix: connection.Id → connection.ID to match hookdeck.Connection.
• pkg/websocket/client.go and client_test.go merged cleanly.

Verified locally: go build ./..., go vet, gofmt clean, and go test -race ./pkg/websocket ./pkg/listen/... passes (the pkg/listen/healthcheck port-443 test fails in my sandbox, but it does so on clean main too — environment-only).

---

Generated by Claude Code

[Copilot]

Pull request overview

This PR improves the reliability and user experience of hookdeck listen against the new server-side websocket proxy by making reconnects resilient to server-side session expiry, ensuring intentional shutdowns close cleanly, and reducing noisy logs for expected server closes.

Changes:

• Send session recreation metadata on every WebSocket connect/reconnect (X-Webhook-Ids, base64-encoded X-Session-Filters) to allow server-side session recreation after Redis TTL expiry.
• Implement clean shutdown behavior (send close code 1000) and reduce log severity for expected close codes (1001 going away, 4001 session expired).
• Add concurrency guards around WebSocket client state and add tests for the new reconnect/close behaviors.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
pkg/websocket/client.go	Adds reconnect headers, clean close on Stop, quieter expected-close logging, and some connection-state synchronization.
pkg/websocket/client_test.go	Adds new unit tests covering session recreation headers, clean close, reconnect quietness, and Stop races.
pkg/listen/proxy/proxy.go	Plumbs connection IDs + filters into the WebSocket client, adds mutex-guarded access for the proxy’s active client pointer, and resets reconnect backoff after successful connects.

Suppressed comments (2)

pkg/websocket/client_test.go:96

• Same issue here: calling client.connect() directly can leave goroutines blocked on notifyClose after shutdown. Use Run() + Connected() instead so the client tears down cleanly.

	if err := client.connect(context.Background()); err != nil {
		t.Fatalf("connect failed: %v", err)
	}
	defer client.Stop()

pkg/websocket/client_test.go:130

• Same here: client.connect() starts pumps but there’s no Run() select loop to drain notifyClose, so a clean close can still leave blocked goroutines. Switch to Run() + Connected() before asserting the close code.

	if err := client.connect(context.Background()); err != nil {
		t.Fatalf("connect failed: %v", err)
	}

	client.Stop()

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[leggetter]

Pushed cf9f0ff with three fixes found while reviewing this branch — two of them user-visible regressions that came in as side effects of otherwise-correct changes.

1. Abnormal closure (1006) was logging at error level. Fixing the latent ws.IsCloseError(err) bug (nice catch — with no codes it always returns false, so every close error went to debug and both error branches were dead) revived the error path for close codes outside the handled set. 1006 falls in that gap, and it's the most common real-world drop: network blip, laptop sleep, LB idle timeout, pod killed without a graceful close. Verified against a server that drops TCP with no close handshake:

ERROR close error: websocket: close 1006 (abnormal closure): unexpected EOF
ERROR If you run into issues, please re-run with `--log-level debug` and share the output ... on GitHub.

That's the opposite of this PR's "quiet reconnects" goal, so 1006 now logs at debug alongside 1001/4001. Added TestAbruptDisconnectReconnectsQuietly, which fails without the fix.

2. Ctrl+C often printed "Connection lost, reconnecting…" on the way out. New behavior from calling Stop() in the signal handler: Stop() closes done, Client.Run takes its <-c.done branch, and that branch closes NotifyExpired. Since Stop() runs before cancel(), the proxy's select sees NotifyExpired ready before (or alongside) signalCtx.Done() and can take the disconnect path, starting the reconnect spinner as the process exits — and that path returns without renderer.Cleanup(), so the spinner is never stopped. A shuttingDown flag set before Stop() disambiguates it. (Checking signalCtx.Err() alone doesn't work — the loop can get there before cancel() executes.)

3. hasConnectedOnce is now an atomic.Bool. Pre-existing, but adjacent to this PR's race-safety work and invisible to -race because pkg/listen/proxy has no tests: it's written by the per-attempt connection-monitor goroutine and read by canConnect() on the Run goroutine. CompareAndSwap also makes the "spawn the health monitor exactly once" guarantee in the comment actually true, now that each reconnect attempt spawns its own monitor goroutine.

go build ./..., go vet, gofmt, and go test -race ./... all clean (the pkg/listen/healthcheck port-443 test fails in my sandbox, but does so on clean main too).

One thing I deliberately left alone: p.currentWebSocketClient().SendMessage(...) at proxy.go:434 and :461 is unchecked while the call site in processEndpointResponse nil-checks. Pre-existing inconsistency and not reachable in practice (the handler only runs once a client exists), so it seemed out of scope here.

---

Generated by Claude Code

[leggetter]

Reviewed locally: go build, go vet, and gofmt are clean, and go test -race ./pkg/websocket/ ./pkg/listen/... passes (including the new client tests). Verified a live connect + clean Ctrl+C shutdown against the current server. LGTM.