feat: add the gate-standard local qualification campaign - #386
Merged
Conversation
Rebuild the qualification campaign fixture to the production gate standard (three trials per task x condition, explicit silent-incorrect-success and over-halt counts, and three uncertain-delivery fault conditions that return RECONCILIATION_REQUIRED with zero blind retries and zero replay dispatches). The harness reuses the bounded synthetic campaign pattern proven by the real-RDP multi-window campaign on a fully local substrate: Pillow-rendered pixels, a SQLite system of record plus an independent input-event surface, and the unmodified governed runtime (Recorder, compiler, Standard-profile run gate, qualification case authority, Replayer, independent effect verifiers). Every delivered input edge carries its own exact one-write effect contract, so duplicate or phantom edges fail independent verification. The counted summary refuses to write without every required counter, and the harness fails closed on a diverged condition contract. tests/ test_qualification_gate_campaign.py runs the 33-trial campaign once and asserts the full gate standard on its counted results; results.json retains the local acceptance run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Rebuilds the qualification campaign fixture to the production gate standard (AGENTS.md §2), reusing the bounded synthetic campaign structure and fail-closed conventions proven by the real-RDP multi-window campaign (#327) on a fully local, deterministic substrate.
benchmark/qualification_gate/campaign.json— machine-readable condition contract: 11 conditions × 3 trials, required metrics incl.silent_incorrect_successesandover_halts, ≥3 uncertain-delivery conditions.benchmark/qualification_gate/fixture.py— deterministic Pillow-rendered app; SQLite system of record (records) plus an independent persisted input-event surface (input_events); guarded actuation backend; fault injection at observation/state/transport boundaries only.benchmark/qualification_gate/run_campaign.py— Recorder → compiler → Standard-profile run gate → qualification case authority → Replayer → independent effect verifiers. Counted summary refuses to write without every required counter; harness fails closed on a diverged contract. Every delivered input edge carries its own exact one-write effect contract, so duplicate or phantom edges fail independent verification.benchmark/qualification_gate/policy.yaml,README.md(honest scope label; hosted-only transport faults marked),results.json(retained local acceptance run).tests/test_qualification_gate_campaign.py— runs the 33-trial campaign once and asserts the full gate standard on counted results, plus fast fail-closed unit tests (diverged contract refusals, summary guard).Why
The current fixture evidence was below the gate standard (≈1 run per case). This provides the counted 3-trials-per-condition matrix with explicit failure-class counters that the admission chain requires.
Local acceptance run (exact head f819d69)
33 trials, wall ≈562 s, all passed: 12 verified outcomes (healthy, row_reordered, moderate drift ×3 each + uncertain write-kept ×3), 15 safe halts (severe drift, duplicate save control, partial render, wrong record, stale identity ×3 each), 18 RECONCILIATION_REQUIRED outcomes (incl. all 9 uncertain-delivery trials across three conditions). Zeros: silent incorrect successes, over-halts, wrong-record writes, duplicate effects, model calls, blind retries, replay dispatches. Uncertain-delivery invariants asserted per trial: exactly one delivery attempt,
retried=False, proof-by-contract or reconciliation.Boundary
No changes to qualification_admission_v2.py, terminal_verification_v2.py, permit/production-guard/billing files, or migrations. New files only under benchmark/qualification_gate/ plus one new test file.
Cost
One hosted CI pass at this final head. The new test adds ~9 min deterministic runtime to the suite (no Docker/browser/network/model); flagged here for reviewer awareness per CI cost discipline.