feat(deletion): add durable whole-community deletion

[kalvinnchau]

Summary

Adds a durable, operator-controlled V1 for deleting an entire Buzz community without deleting another tenant's data.

The workflow is exposed through buzz-admin deletions:

• sweep records independent fleet storage-taxonomy observations
• submit, list, inspect, and approve manage a deletion request
• unblock resumes a fail-closed request after an operator records remediation identity and reason
• run and drain execute bounded work

Requests advance through a PostgreSQL-backed state machine and stop at retention_pending after logical deletion has been independently verified across PostgreSQL, object storage, and Redis.

This PR ships the engine and CLI, not a continuously running worker or Kubernetes packaging. For V1, a cluster/VM administrator invokes /usr/local/bin/buzz-admin from the existing relay image, for example with kubectl exec or an equivalent container/VM exec path.

What whole-community V1 removes

For the target community, V1 removes:

• rows from the allowlisted community-scoped PostgreSQL catalog, including members, profiles, authored events and bodies, DMs, reactions, mentions, memberships, tokens, workflows, moderation, audit, feedback, and rate-limit state
• media sidecars and upload-attribution records under _meta/<community>/ and _uploads/<community>/
• Git repository pointers under repos/<community>/
• Redis keys under buzz:<community>:*

The community row survives as a permanent tombstone, and deletion control-plane records remain as evidence of the request, approval, execution, and result.

Safety model

Deletion is not a broad DELETE CASCADE followed by optimistic cleanup. The destructive boundaries are durable and fail closed.

1. Inventory and approval

• submit resolves the target and freezes the schema plus summary-only storage inventory.
• Approval is bound to the exact request, community, and frozen inventory digest.
• Unsupported manifest versions, malformed keys inside the target's owned prefixes, live scoped-table/write-fence coverage drift, frozen-inventory mismatch, and approval mismatch block execution rather than guessing. Migration and catalog revision numbers are not authorization gates; the executor validates the live safety shape instead.
• Storage inventory is server-side prefix scoped to exactly:
  • _meta/<community>/
  • _uploads/<community>/
  • repos/<community>/
• The deletion path never lists the whole shared bucket and has no arbitrary per-community object cap. Its listing work is proportional to the target community's bindings, not total fleet storage.
• Fleet-wide taxonomy sweeps remain independent observability. They report unknown writer shapes but do not gate deletion submission, fencing, or destructive progress. Maintainers must add deletion taxonomy coverage whenever a new community-owned object-key class is introduced; writer-coverage tests bind the current media and Git writers to that contract.

2. Quiesce, fence, and destructive freeze

• Writes continue through submission, inventory, and approval. They stop when execution moves the target into quiescing and then establishes the durable fence.
• Already-admitted external effects finish under heartbeated serving-write leases; the exact admitted lease may renew while the community is quiescing, but new lease acquisition is rejected. The executor drains admitted leases before destructive work.
• Invite minting after quiescing begins fails as typed AccessDenied (HTTP 503 at the relay boundary) before an invite can be persisted.
• Database triggers enforce the community write fence across the complete catalog of community-scoped tables. Startup/readiness and destructive execution validate that catalog so a newly added but unfenced table cannot silently escape.
• Named isolation assumption — fresh write snapshot. Every writer transaction that can reach a community-fenced relation must use PostgreSQL READ COMMITTED; each guarded write therefore observes a statement snapshot no older than acquisition of the community deletion lock. REPEATABLE READ and SERIALIZABLE can retain a pre-fence snapshot and are unsupported for writers. The writer pool refuses non-READ COMMITTED sessions at connection setup, and both SQL fence functions reject an explicit per-transaction isolation override with SQLSTATE 25000. Configuration-delivered bad isolation can surface through SQLx as a pool-acquire timeout because every after_connect attempt is rejected; the precise community writes require READ COMMITTED isolation reason remains observable when the SQL guard is reached. Read-only replica transactions are outside this assumption.
• Holding the shared advisory lock until the guarded write executes is a separate liveness condition: under READ COMMITTED, releasing it early does not permit resurrection because the trigger rechecks the fence, but it can turn a fleet sweep into a statement-wide SQLSTATE 55000 abort.
• After the fence closes writers, storage is re-enumerated into chunked side-table rows. Per-prefix counts and digests bind those concrete keys to the destructive manifest.
• Manifest chunk insertion, update, and deletion are protected after freeze. This closes the race where an unbound key could otherwise appear after the manifest was committed.

3. Checkpointed destruction

• Target-owned object bindings are deleted from the frozen destructive manifest in bounded batches with durable progress.
• The concrete key list lives in chunked side-table rows rather than one request-row JSON value. It supports large communities, resumable execution, and terminal cleanup.
• Missing objects are accepted as idempotent crash-window outcomes; malformed ownership, changed evidence, and unexplained target-prefix drift fail closed.
• PostgreSQL purging remains scoped by community_id, including the guarded NIP-RS hard-delete path discovered with real Desktop kind 30078 read-state data.
• Redis cleanup explicitly scans and UNLINKs only buzz:<community_id>:*. Natural expiry is insufficient because some keys, including tunnel generation counters used as fencing state, are deliberately persistent.

4. Independent verification

• PostgreSQL logical absence is checked after purge.
• The three target-owned storage prefixes are freshly inventoried again and must be empty.
• Redis requires two complete empty namespace scans.
• Only after all three stores pass does the request advance through logically_verified to retention_pending.

What V1 deliberately does not erase

Shared content-addressed storage

Per-community deletion removes bindings, metadata, attribution records, and Git pointers. It does not physically delete fleet-shared CAS bytes that another community may still reference:

• media blobs and thumbnails
• Git manifests, packs, and indexes (manifests/, packs/, and idx/)

Safe reclamation requires a separate fleet-wide reachability and retention GC. Unknown keys elsewhere in the shared bucket do not block one community's deletion; malformed or unrecognized keys inside that community's three owned prefixes still fail closed.

External retained copies

The online logical-deletion proof does not erase object versions/replicas, database backups/WAL, CDN copies, provider retention copies, or observability exports. Those require their own retention and purge controls.

Member-only erasure

This PR erases a whole community. It does not implement the different operation "erase one npub while preserving the community."

Removing membership or accepting NIP-09 is not member erasure. A member-only workflow would need to find and selectively remove or redact authored event content and pubkeys, profile data, DMs, reactions, mentions, memberships/roles, tokens, workflows/subscriptions, upload attribution, moderation/audit history, repository attribution, and identity embedded in tags or JSON. It would also need explicit rules for ownership transfer, surviving replies and thread metadata, audit-chain integrity, immutable Git history, and shared-CAS reachability. That requires a pubkey-level fence and selective graph rewrite; it is a separate deletion product, not a safe extension of this whole-tenant worker.

In scope

• migration 0029_community_deletion.sql: requests, approvals, leases, manifest chunks, checkpoints, tombstones, and the universal write-fence catalog
• durable executor leases, generations, heartbeats, retry/block state, and resumable stage transitions
• operator-driven sweep, submit, list, inspect, approve, unblock, run, and drain commands
• serving-path fences for database writes and external effects across event ingest, media, Git, workflow, push, invites, mesh/tunnel, and related paths
• target-prefix-only storage inventory, summary manifests, post-fence destructive chunks, and bounded batch deletion
• exact community Redis namespace purge and two-pass absence verification
• cross-community isolation, crash/resume, manifest-integrity, writer-taxonomy, and schema/migration regressions
• desired-state schema/schema.sql support without requiring a SQLx migration ledger

Deferred / not covered

• dedicated Helm/chart worker Deployment, service account, secrets, probes, resources, and network policy
• autonomous buzz-admin deletions worker poll loop and worker-only health server
• least-privilege separation among migration, relay-serving, and destructive execution roles
• fleet-wide shared-CAS physical GC
• backup/provider/CDN/observability retention completion
• member-only erasure
• provider-native conditional-delete improvements
• a general force-continue escape hatch; permanent safety failures remain fail closed unless an operator remediates the cause and records an audited unblock

The removed continuous-worker implementation remains deferred; no remote follow-up branch is claimed by this PR.

Validation

Current PR head and repository state

Current pushed head: 359d8402ee15f049768f54156f67b953c7a7e2ed, rebased onto cc9a2f783375e51a6e8d1f2f9d01d5f7e22813d1 (origin/main at push time). The complete PR diff is now 47 files, 9,834 additions, and 517 deletions.

The bespoke source-scanner stack was removed to keep this PR scoped to community deletion. Tyler/team requested the underlying fenced-write safety behavior, not ast-grep, crates/buzz-db/tests/community_fenced_writes.rs, its 27 fixtures, or the new scripts/lints/community_*.yml rules. Those scanner-specific files, dependencies, Hermit links, and runner wiring are absent from the current tree. The production database write fence, startup/destructive live-catalog validation, and deletion behavior remain.

Source validation on this exact SHA passed:

• cargo fmt --all -- --check
• bash -n scripts/run-tests.sh
• cargo nextest run -p buzz-db --all-targets: 102 passed, 173 skipped, 0 failed
• cargo nextest run -p buzz-deletion --all-targets: 10 passed, 9 skipped, 0 failed
• cargo nextest run -p buzz-admin --all-targets: 1 passed, 0 failed
• affected-package/all-target Clippy with warnings denied
• lockfile consistency
• Helm 3.16.4 lint and all 44 chart unit tests
• Helm region controls using that fixture: default BUZZ_S3_REGION=us-east-1, explicit eu-west-2 override, and blank-region schema rejection

The prior Kubernetes battery below was run against 928992237358a3294621ac0280830b77155abc04. It remains useful evidence for the patch-equivalent production deletion implementation, but it is not claimed as exact-SHA evidence for current head 359d8402ee15f049768f54156f67b953c7a7e2ed; the current cleanup removes only scanner/test/tooling infrastructure. CI restarted for the new head after the rebase and is pending. Human review remains CHANGES_REQUESTED.

Prior-head live Kubernetes deletion and safety gates

The full program used one immutable image, real PostgreSQL, Redis, MinIO, and a three-relay Kubernetes release:

• source: 928992237358a3294621ac0280830b77155abc04 (prior head)
• image: buzz-e2e:sha-928992237358
• immutable image digest: sha256:a1a204f4618ac22d9e210be5e5290645a15d79831ae30b0e44379357c8e4a895
• evidence root: /tmp/buzz-e2e/20260807T033025Z-928992237358-full-gates/
• evidence-manifest digest: 82875c5bc9bea7370b796a7aef3457b3a1c8306c84c59e0f7388bbb5ad30e865

Passed gates at that prior head:

• Chart/operator region: default us-east-1, explicit nondefault propagation, blank-region schema rejection, live in-pod environment, and an in-pod taxonomy sweep over 18 objects with zero unknown.
• Fenced writers and lifecycle: open-write/fence ordering; 100-attempt anti-starvation; invite, push matcher, and exhausted-reaper bystander isolation; non-READ-COMMITTED rejection; manifest/tombstone contracts; eight-failure stage block and audited unblock.
• Destructive lifecycle: submit → approve → run → retention_pending; PostgreSQL tombstone and Redis/S3 verification true; zero retries/errors; terminal reruns rejected with exit 5.
• Fresh 10,001-object crash boundary: exactly two chunks (10,000 + 1). The executor deleted chunk 0 from MinIO while its PostgreSQL stamp was row-lock-blocked, was killed with SIGKILL, left one object and both stamps absent, then resumed the same request under generation 2 to zero objects and terminal state.
• Independent dead-owner recovery: a dedicated executor claimed generation 1, blocked before effects, and was killed through containerd with SIGKILL (no TERM cleanup). The request remained owned and unreclaimable before lease expiry; a successor claimed generation 2 after 60 seconds and completed with two attempts and zero retries.
• Three-pod socket isolation: ordinary NIP-42 and joined huddle-audio target witnesses on every replica received exact 1008 / community deleted; healthy-tenant witnesses on those pods remained live; deleted-host reconnect returned HTTP 404.
• Health/provenance: all replicas independently returned ready and retained the exact image digest before/after destructive runs and an audio-enabled rolling restart; PostgreSQL, Redis, and MinIO were healthy at close.

Instrument corrections were retained as evidence rather than counted as product failures: a foreground PostgreSQL forward caused an initial PoolTimedOut; Kubernetes pod deletion exercised graceful TERM rather than dead-owner recovery; shell-background socket witnesses died with their parent; and the first image build hit the corporate TLS proxy. Detached forwarding/witnesses, containerd SIGKILL, and the configured internal CA/Artifactory mirror produced the discriminating runs without weakening product security.

Prior-head cleanup

For the prior-head Kubernetes run, the Helm release was removed, namespace absence was verified, run-owned Screen sessions were absent, and that source worktree remained clean. The evidence manifest was independently recomputed and every indexed artifact passed shasum -a 256 -c. The current 359d8402 source worktree is also clean after the scanner-only cleanup and push.

[wesbillman]

Review submitted by Carl on Wes's behalf. I traced the deletion lifecycle, DB trigger and lease model, storage manifest freeze/delete path, Redis purge/verification, and serving-side call sites, with independent adversarial passes from Mongo and Princess Donut. CI is green, but these findings block the feature's advertised safety/operability contract.

[wesbillman]

[P0] Bump this to 28 and pin it to the actual migration set. This PR moved community deletion to 0028_community_deletion.sql, but SQLx-backed deployments report max(version) = 28. validate_destructive_migration_version requires exact equality, so validate_catalog() rejects every submit/inventory and execution attempt with expected 27, got 28. Desired-state deployments hide this because they have no _sqlx_migrations ledger. Please also add a regression that compares this constant to the migrator's concrete max version rather than testing only values derived from the constant.

[wesbillman]

[P1] Keep already-admitted leases renewable while quiescing, or otherwise preserve the exclusion proof for long effects. begin_quiescing promises that admitted external effects finish and the fence drains them, but this predicate makes the next 10-second heartbeat fail as soon as state becomes quiescing; ServingWriteGuard then cancels the protected operation and its drop path releases the lease. Media/video and Git object-store writes can legitimately exceed that interval, and cancellation of the client future does not prove the remote S3 mutation did not commit after the lease was released. That allows destructive listing/freeze to overtake an admitted remote write—the exact race the lease is meant to close. New acquisition is already blocked by deletion_state = 'active'; renewal should remain allowed for the exact existing lease/fence generation through quiescing, with a regression covering a long admitted external write across quiesce.

[kalvinnchau]

Fixed in 4e0726e5a. Renewal now permits only the exact existing, unexpired lease while the community is active or quiescing; owner, lease generation, fence generation, expiry, tombstone, and community fence-generation checks remain intact, while new acquisition is still restricted to active. Added PostgreSQL coverage for renewal/drain behavior and a long-running ServingWriteGuard heartbeat regression across quiesce. Both pass against isolated PostgreSQL at PR HEAD 7c755960d.

[wesbillman]

[P2] This does not map the actual fence failure. mint_relay_invite performs an ordinary insert; the new relay_invites trigger raises SQLSTATE object_not_in_prerequisite_state, which arrives as DbError::Sqlx, not DeletionSafety. The fallback therefore returns HTTP 500 during quiescing/fencing. Guard the invite transaction so the boundary produces a typed access/fence error (and map it to the intended restriction status), or explicitly translate this SQLSTATE. Please cover mint-after-quiesce at the API boundary.

[kalvinnchau]

Fixed in 7c755960d. mint_relay_invite now takes the community advisory lock and checks the write fence inside the same transaction before inserting, returning typed DbError::AccessDenied instead of leaking the trigger's SQLx error. The API maps that typed fence to HTTP 503. Added PostgreSQL coverage proving mint-after-quiesce returns AccessDenied and persists no row, plus an HTTP boundary regression proving the typed error maps to 503. Both pass at PR HEAD 7c755960d.

[kalvinnchau]

All three original blocking review findings are addressed on current head 928992237358a3294621ac0280830b77155abc04: migration-revision coupling was removed in favor of live structural validation; exact admitted serving leases remain renewable through quiescing while new acquisition is blocked; and invite minting now takes the lifecycle guard and maps its typed AccessDenied result to HTTP 503 with database/API regressions. The latest repair also restores the calibrated fenced-writer class guard, fixes Helm-deployed deletion operator S3-region operability, and corrects the PR isolation/evidence documentation. Existing CI is running on this head, and the independent exact-SHA Kubernetes battery is in progress. @wesbillman please re-review when those checks are terminal.

[tlongwell-block]

Blocking review feedback at head 8a4bf860bf0fd555f73711b52140f2622436066f — posted by Eva (agent) on Tyler's behalf, consolidating this round: three independent source reviews (Eva, Wren, Dawn) + two fresh-stack live E2E runs (Max).

First, credit: the destructive path holds. Across all reviews and live runs we found no way to leak tenant data, hit a bystander, or bypass the fence — the invariants-in-the-database design, the two-sided validate_catalog vs missing-only validate_serving_catalog asymmetry, and the digest-bound two-person approve are genuinely strong. The block is on an orthogonal axis: a correctly-targeted deletion that fails mid-flight has no exit, and the stranded state is a permanent tenant-wide write outage.

Demonstrated failure mode (live, fresh stack)

1. Submit + approve + fence a community (deletion_state='fenced', all 30 scoped tables rejecting that tenant's writes).
2. Any ordinary migration ships that adds a table with a community_id column while the deletion is in flight.
3. Next run: claim() calls validate_catalog() first (crates/buzz-db/src/deletion.rs:961) → DeletionSafety("catalog drift (missing=, unknown=<table>)") → classified permanent by is_permanent_error (crates/buzz-deletion/src/lib.rs:332) → never retried, and the claim never succeeds.
4. Because the failure fires before the claim exists, nothing reaches block(): the request sits at stage=fenced, blocked_at=NULL, last_error=NULL. deletions unblock refuses with "is missing or is not blocked". Live receipt: run rc=5 ... catalog drift (missing=, unknown=max_drift_probe); target writes rejected (community write fenced ... generation 1), paired bystander control writes committed.
5. No code path returns a community to active. All three production writes to deletion_state are forward-only (deletion.rs:1129/1192/1580); the Command enum (lib.rs:219-283) has no abort/restore. A DBA's plain UPDATE ... SET deletion_state='active' is rejected by the tombstone trigger; the only working manual exit is forging the executor GUCs in psql — which does not bump the fence generation, so a still-leased executor remains authorized to resume destroying the tenant that was just rescued.

Also demonstrated: S3 versioning enabled between submit and drain → permanent post-fence failure with the same no-exit shape; and the consecutive_retries >= 8 cap (deletion.rs:1785) converts sustained transient post-fence failure into a durable block.

Required to clear the block

(a) An abort command. Legal only while deletion_state IN ('quiescing','fenced') and no BindingsRemoved checkpoint exists. It must restore active, bump deletion_fence_generation (every executor and serving lease is already generation-bound, so stale holders fail closed against the new value — this is what makes it safe), and terminally close the request so it cannot be re-claimed (otherwise an aborted request at a bumped generation is re-claimable and the next run re-fences the rescued community). Post-BindingsRemoved there is no honest abort and it must refuse loudly. Note this is not new machinery: set_executor_gucs + generation bump is how every legitimate transition already works; this is the audited, checkpointed version of what an operator is currently forced to do by hand.

(b) Claim-time validate_catalog failures must leave a durable, inspectable record (last_error/blocked_reason or equivalent) so inspect can explain a stranded request. Today the cause exists only in stderr of a command someone already ran; an operator paging mid-incident sees a request indistinguishable from one healthily waiting for an executor.

(c) product_feedback classification conflict — recorded decision needed (fast-follow acceptable). _operator_global_tables declares it operator-global, "community_id is provenance only" (schema/schema.sql:837, migrations/0017_product_feedback.sql:23), but it is in EXPECTED_SCOPED_TABLES/PURGE_SCOPED_TABLES (deletion.rs:59/96) and gets a fence trigger. The purge set is the newer claim and silently overrides an explicit prior "provenance only" declaration — validate_catalog never consults the registry. Either drop the registry entry or exclude the table from purge (e.g. retain rows with community_id nulled), but the choice should be on the record.

A runbook alone does not clear (a): the trigger condition is a routine deploy landing during a deletion window (which, for a large tenant, spans full object-store enumeration and bulk deletes), and the blast radius is a full write outage for a live tenant with no sanctioned recovery.

Happy to re-review as soon as the recovery path lands — the rest of this PR is ready.

[tlongwell-block]

Addendum to the blocking feedback above (same review team, same head 8a4bf860bf0fd555f73711b52140f2622436066f): one new defect found by the live acceptance battery, and a hardened spec for the abort command after adversarial review of our own proposal. Every claim below was independently re-verified in source at the exact head.

(d) NEW: schema/schema.sql omits community_deletion_manifest_keys

Migration 0029_community_deletion.sql:159 creates it; schema.sql carries the other four deletion control-plane tables (community_deletion_requests/_approvals/_checkpoints/_executor_heartbeats, lines 1130-1284) but not this one. Any fresh desired-state install (e.g. pgschema apply --file schema/schema.sql, which scripts/start-isolated-test-relay.sh uses) gets a deletion engine that fences successfully and then fails when freeze_destructive_manifest calls clear_manifest_key_chunks:

stage=fenced; retry_count=1
last_error=... relation "community_deletion_manifest_keys" does not exist

Live receipt on a fresh stack; the same battery passes fully when the DB is initialized via buzz-admin migrate, so this is pure desired-state drift, not a code defect. Combined with finding (a) — no abort — a fresh-schema deployment's first ever deletion strands the tenant fenced. One-line fix: add the table (and its indexes) to schema.sql.

Hardened spec for the abort command (supersedes the (a) details above)

We red-teamed our own proposed spec; three necessary corrections:

1. Abort boundary is Approved|Fenced request stages only — refuse Drained and later, regardless of checkpoints. "No BindingsRemoved checkpoint" is not a safe gate: Drained deletes S3 chunks one at a time (buzz-deletion/src/lib.rs:1089-1136) and only checkpoints BindingsRemoved after the last chunk, so a request at stage=drained with no checkpoint may already have physically destroyed tenant data. No honest abort exists there.

2. Generation bump + terminal close is not sufficient against a concurrently running executor; forward transitions must re-verify the lease after acquiring the advisory lock. begin_quiescing verifies the lease, then waits for pg_advisory_xact_lock, then re-reads the generation FOR UPDATE and sets executor GUCs to match whatever it finds (buzz-db/src/deletion.rs:1116-1152) — so an executor paused at the lock when abort commits will read the bumped generation, satisfy the tombstone trigger, and put the rescued community straight back into quiescing (its UPDATE ... WHERE deletion_state IN ('active','quiescing') matches). The abort transaction must take the same advisory lock and atomically terminalize the request + invalidate the lease + restore the community, and begin_quiescing/fence must re-run lease verification after lock acquisition (or predicate their community UPDATE on current request lease/stage). A regression test should pause begin_quiescing between verification and lock, commit an abort, and prove the paused executor cannot proceed.

3. "Restore active" must restore the pre-quiesce archived_at. begin_quiescing writes archived_at = COALESCE(archived_at, now()) (deletion.rs:1129-1132), and serving paths gate on archived_at IS NULL (e.g. buzz-db/src/lib.rs:1254). Restoring only deletion_state leaves the tenant dead-in-effect; clearing unconditionally is also wrong because submit() does not reject already-archived communities (deletion.rs:539-569). Capture the pre-quiesce value durably (or make submit reject archived targets) and restore it atomically with lifecycle + generation.

Plus two spec clarifications: the terminal state must be an explicit aborted representation excluded from both claim and unblock (the stage CHECK at migrations/0029_community_deletion.sql:19-27 has no such value today, and unblock deliberately re-arms runnable stages), with an audit checkpoint recording actor/reason and old/new generation; and for the quiescing edge, admitted serving leases bound to the current generation must either drain before the bump or have a defined completion policy, so abort doesn't strand their in-flight external effects.

[tlongwell-block]

Second addendum (same team, same head 8a4bf860bf0fd555f73711b52140f2622436066f): finding (d) is wider than first reported, and finding (b) needs to be widened from one call site to a class. Independently derived by a second reviewer via a full both-paths provisioning diff, replicating and extending the earlier live result; every claim re-verified in source by the verdict owner.

(d) widened: schema.sql is missing TWO tables from 0029, plus the manifest immutability guard

Applying both provisioning paths to isolated Postgres 17 containers and diffing pg_tables: the migration path yields 60 tables, the pgschema apply --file schema/schema.sql path yields 58. Missing from schema.sql:

• community_deletion_manifest_keys (0029:159) — as reported above, plus
• storage_taxonomy_sweeps (0029:225) — used at runtime by the Sweep command (buzz-db/src/deletion.rs:1421, :1457), so desired-state deployments also break the taxonomy sweep, and
• CREATE FUNCTION protect_community_deletion_manifest_keys + its community_deletion_manifest_keys_guard trigger (0029:174/217) — the immutability guard on frozen key chunks.

Reverse direction is empty; the other 5 tables and all other functions from 0029 match in both paths.

Worth stating the failure anatomy precisely, because every gate passes on the way in: submit/approve/fence never touch the missing tables; validate_catalog passes (30/30 — the missing tables carry no community_id, so tenant-catalog validation structurally cannot see control-plane absence); DeletionStore::ping returns true (it probes only community_deletion_requests, and its comment at deletion.rs:527-529 explicitly claims pgschema "creates the same deletion objects" — the comment is currently false). Then Fenced hits DELETE FROM community_deletion_manifest_keys (deletion.rs:1297) → raw sqlx::Error → classified transient → retried until consecutive_retries >= 8 converts it to a durable block. Fenced tenant, hard-down, recorded reason "relation does not exist," no recovery path (finding (a)).

Fix: add both tables, the guard function, and the trigger to schema.sql. Suggested hardening: the existing hand-written desired_schema.contains(...) assertions (buzz-db/src/migration.rs:910 region) were never extended to 0029; a real equivalence test — apply both provisioning paths, diff the catalog — would have caught this and will catch the next one.

(b) widened: the invisible-failure class is the whole pre-claim path, not validate_catalog

Everything from process start to a committed claim() row runs without a lease token, and both durable-record paths (block() at deletion.rs:1902, record_retry() at :1772) require one via verify_lease. Enumerated members, all sharing the blocked_at=NULL / last_error=NULL / unblock refuses signature:

1. connect_services() / required_env (buzz-deletion/src/lib.rs:515, :553) — missing/empty BUZZ_S3_*, REDIS_URL, DATABASE_URL aborts before a DB handle exists.
2. validate_catalog() at claim() (deletion.rs:961) — already reported.
3. claim() returning None (lib.rs:776-780) — the bail message cannot distinguish not-runnable / blocked / leased-elsewhere, and records none of them.
4. row_to_request / DeletionStage::from_str (deletion.rs:2570, :190) — unknown stage string is a DeletionSafety from inside the claim query's own row mapping.

The fix shape should therefore be: the pre-claim path records failure against the request (where a target request is identifiable), rather than instrumenting the one known call site.

Related observability note: buzz-deletion contains no tracing/log macros at all (only println! JSON on success paths), so process stderr is the entire failure-observability surface today. Even minimal tracing::error! coverage on the failure paths would materially improve incident response independent of the durable-record fix.

[kalvinnchau]

Addressed the merge-blocking retry defect at exact head fe8c147e53ac6b0721543e12ba0349600604eebe.

What changed

block_preclaim_setup() now uses a retry-safe checkpoint upsert on the unique (request_id, stage, unit_key) key instead of a plain insert. On a repeated pre-claim setup failure it now atomically:

• restores the request block;
• keeps the checkpoint at status = 'failed';
• updates lease_generation, bounded error, and detail to the current failure;
• increments attempts;
• clears completed_at.

This matches the existing checkpoint_failed_tx() conflict behavior while preserving checkpoint history. Request re-blocking and checkpoint persistence remain in the same transaction, so an incomplete repair cannot leave the request runnable and apparently healthy.

The PostgreSQL regression now covers the full supported sequence:

block → operator unblock → same-unit block again

It verifies the request is re-blocked, the latest error/detail are durable, attempts == 2, and the failed checkpoint remains incomplete.

Verification at this exact head

• PostgreSQL 17 live store regression: passed.
• Actual buzz-admin deletions run → unblock → run reproduction: passed; the second failure exited nonzero and re-blocked atomically.
• All 20 deletion PostgreSQL tests: passed.
• buzz-db: 104 passed.
• buzz-deletion: 10 passed.
• Clippy with warnings denied, formatting, and diff checks: passed.
• PR CI: 36/36 terminal non-failing (30 success, 6 skipped), 0 failed/pending.

I also ran deployment-shaped validation using one immutable candidate image (sha256:8dea1868bcca115aa7f20ef0369e43544a155ef1c02ead2f158909b06cc9e4bc):

• isolated Docker Compose with PostgreSQL 17, Redis 7, MinIO, and the candidate relay: retry/re-block path and complete deletion lifecycle passed; target data was removed and bystander data survived;
• Docker Desktop Kubernetes with three relay replicas and bundled shared stores: deletion reached retention_pending; all 3/3 replicas remained Ready on the exact candidate digest, and bystander data survived.

Independent final review reported no findings at this exact head. Local branch, remote branch, GitHub PR head, and fetched PR ref all resolve to fe8c147e53ac6b0721543e12ba0349600604eebe.