Skip to content

flagd: align retry defaults with spec, fix retry maxAttempts, emit STALE on stream errors #408

Description

@toddbaert

We must implement open-feature/flagd#2027, which proposes changing the spec defaults so the stream-reconnect backoff can't leave us disconnected longer than the stale grace period.

  • config.py:31 DEFAULT_RETRY_BACKOFF_MAX: 12000 -> 5000
  • config.py:32 DEFAULT_RETRY_GRACE_PERIOD_SECONDS: 5 -> 10

Both resolvers also set maxAttempts to 3 (resolvers/grpc.py:105, resolvers/process/connector/grpc_watcher.py:82), but the spec's retry policy specifies 4 (the initial attempt plus retries at 1s, 2s, 4s); should be 3 -> 4. Harmless to fix now since nothing clamps at a 5000 cap.

Separately, we don't emit STALE on sync-stream errors. _state_change_callback (grpc_watcher.py:178) only emits STALE when the gRPC channel enters TRANSIENT_FAILURE, and a stream-level error doesn't change channel state, so _handle_rpc_error (:273) logs at debug and we silently wait retry_backoff_max_ms in _wait_before_reconnect (:318) before re-establishing. For that whole window we're disconnected while still reporting READY.

Measured on 0.5.2, in-process resolver, flagd latest:

Scenario Result
stream_deadline_ms=3000, flagd untouched stream dies at 3.01s, reconnects at 15.01s, zero lifecycle events
default deadline, flagd killed for 1.3s zero lifecycle events

With the default stream_deadline_ms of 600000 that's a silent 12s disconnect every ~10 minutes. Java, Go and JS all emit STALE on stream error here, and per the spec a stream disconnect should emit STALE, then ERROR after retryGracePeriod.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions