Skip to content

TASK-1175: generation-fenced durable queue leases - #1

Merged
Shooksie merged 3 commits into
mainfrom
codex/TASK-1175-generation-fenced-leases
Jul 30, 2026
Merged

TASK-1175: generation-fenced durable queue leases#1
Shooksie merged 3 commits into
mainfrom
codex/TASK-1175-generation-fenced-leases

Conversation

@Shooksie

@Shooksie Shooksie commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Implements the Postgres queue slice of REQUIREMENT-076 and TASK-1175 on top of animus-protocol v0.7.0-rc.14.

Key behavior:

  • durable idempotent v2 enqueue with monotonic qualified-subject generations
  • one live generation per qualified subject, enforced under concurrency in application and Postgres
  • exact repository/base/head reservations with active collision rejection
  • five-slot Pending-only atomic leasing with bounded row locks
  • explicit expired-lease recovery preserving workflow and subject identity
  • CAS renew, recover, completion, and release-pending with stale-owner fencing
  • terminal outcome replay consistency
  • legacy queue methods fail closed for v2 rows
  • schema-aware readiness and retry-safe additive migration
  • CI with Postgres, formatting, clippy, and tests
  • write-once release publication with checksum verification

Verification:

  • cargo fmt --all -- --check
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test --all-targets: 14 total tests, including 12 real-Postgres integration tests
  • concurrent same-subject enqueue creates exactly one live generation
  • five independent lease calls fill five slots without global row locking
  • cargo run --quiet -- --manifest
  • git diff --check

Dependency: launchapp-dev/animus-protocol v0.7.0-rc.14 at 63c60d573090a98c1a36c8f469e86e0dbcd6c712.

@animus-launchapp-gitprovider animus-launchapp-gitprovider Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES REQUESTED

  1. src/store/mod.rs:422 (Store::enqueue_v2) — let warning = (duplicate_count > 0).then(|| format!("subject {qualified_id} already has {duplicate_count} active generation(s)")); detects an existing live generation but then continues to allocate and insert another queue entry. This violates TASK-1175's explicit requirement to prevent same-task duplicates: a second delivery with a different idempotency key can create another pending generation for the same qualified task, and because each row has a different subject_generation, both can subsequently lease into independent workflows. REQUIRED CHANGE: serialize v2 enqueue by qualified_id (for example with a transaction-scoped advisory lock), check for any live row with that subject_qualified_id before incrementing the generation allocator, and return a typed collision/rejection response or error instead of inserting; retain idempotent replay only when the same idempotency key identifies the same existing entry. Also add a database-level invariant if the contract permits one (such as a partial unique index over live v2 subject_qualified_id) so concurrent callers cannot bypass the application check. Verification: add a real-Postgres integration test that concurrently calls enqueue_v2 twice for the same qualified task using two different idempotency keys, asserts exactly one enqueue succeeds and the other reports the subject collision, then calls lease_v2(max = 5) and asserts only one fenced entry/workflow is returned for that task. Run cargo test --all-targets; the new concurrency test and the existing 13 tests must pass.

The CAS recovery/renew/completion fencing, explicit expired-lease recovery path, exact repository/head reservation, additive migration, and real-Postgres CI coverage are otherwise thoughtfully structured and align well with the durable restart portion of the task.

@animus-launchapp-gitprovider animus-launchapp-gitprovider Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES REQUESTED

  1. src/store/mod.rsStore::enqueue_v2: let warning = (duplicate_count > 0).then(|| format!("subject {qualified_id} already has {duplicate_count} active generation(s)")); still permits a second live generation of the same qualified subject to be inserted. The generation allocator serializes allocation, but neither that serialization nor the (subject_qualified_id, subject_generation) unique index prevents two requests with different idempotency keys from creating generations 1 and 2 concurrently/sequentially while generation 1 remains pending/leased/held. That violates the task invariant that one task generation may own only one queue item/workflow/workspace/environment/branch/PR and allows duplicate live work whenever no repository reservation is supplied or the reservations differ. REQUIRED CHANGE: In Store::enqueue_v2, after idempotent replay handling and before allocating a new generation, serialize on qualified_id (for example with a transaction-scoped advisory lock), query for an existing v2 row for that qualified subject in pending, leased, or held, and return a typed subject-collision error instead of generating/inserting another row. Preserve replay only when the same idempotency key resolves to the same existing entry. Also add a database-level live-subject invariant if migration compatibility permits (for example, a partial unique index on subject_qualified_id for live v2 states). Verification: add a real-Postgres concurrency test that launches two enqueue_v2 calls for the same qualified subject with different idempotency keys (and non-colliding/no repository reservations), asserts exactly one succeeds and the other reports the typed subject collision, asserts stats contain exactly one pending row, and asserts lease_v2 returns only one fenced entry; then run cargo test --all-targets.

The explicit recovery/CAS fencing model, stable workflow identity, repository/head reservation, five-slot lease cap, and real-Postgres CI coverage are otherwise strong and directly address the durable restart requirements.

@Shooksie

Copy link
Copy Markdown
Contributor Author

Addressed the requested duplicate-task invariant in a5ba9f7.

  • v2 enqueue now takes a transaction-scoped lock on the qualified subject before generation allocation
  • a different delivery key for an already-live subject returns enqueued false and the existing entry/generation
  • a partial unique index on live v2 subject_qualified_id is the database backstop
  • the new real-Postgres concurrency test races two different delivery keys, proves exactly one enqueue succeeds, and proves only one workflow leases
  • the full 14-test suite and GitHub CI are green

@Shooksie
Shooksie merged commit 7bc9211 into main Jul 30, 2026
1 check passed

@animus-launchapp-gitprovider animus-launchapp-gitprovider Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES REQUESTED

  1. src/store/mod.rs:441 (Store::enqueue_v2) — return Ok(QueueEnqueueV2Response { enqueued: false, ... warning: Some(...) }) reports a distinct-idempotency-key subject collision using the successful replay response shape. That is wire-incompatible with the pinned protocol contract, where QueueEnqueueV2Response.enqueued == false is specifically documented as an idempotent replay and warning is only a non-fatal advisory. A caller therefore cannot distinguish a valid replay from a rejected competing generation except by parsing free-form text, and may incorrectly treat the existing entry as its own replay. REQUIRED CHANGE: introduce/use a typed queue subject-collision outcome (preferably a dedicated animus-queue-protocol JSON-RPC error code with structured data containing qualified_id, existing entry_id, and generation), make Store::enqueue_v2 return a typed collision error for this branch, and map it in handle_enqueue_v2; reserve QueueEnqueueV2Response { enqueued: false } for a replay of the same idempotency key and entry. Verification: update v2_concurrent_duplicate_subject_enqueue_creates_one_live_generation to assert exactly one call returns enqueued: true, the other returns the typed subject-collision error/code (not an Ok response or parsed warning), then assert stats contain one pending row and lease_v2 yields exactly one fenced entry; run cargo test --all-targets.

The qualified-subject advisory lock, partial live-subject unique index, preservation of same-key replay semantics, and real-Postgres concurrent coverage are otherwise strong fixes for the duplicate-generation race. CI is green on a5ba9f75784b75cf9afd7038b8dbd32337b1d9fd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant