Feat/composable scenarios - #3
Merged
Merged
Conversation
The genmodel const dropped an internal `.agent/SWEEP-BACKLOG.md` reference from the Law-7 comment, but the golden fixtures were not regenerated, so the byte-diff golden tests (every_scenario_matches_its_golden, helix_const_matches_golden_block, tracked_cb_basic_config_stays_in_sync) failed against a scrubbed generator. Propagate the scrub to the fixtures. Full suite green (279 tests).
A test scenario was one hardcoded enum variant mapped to one frozen golden fixture; features (ws, timing-games, extra-validation, sigverify, min-bid, topology, client pair, signer) could not be combined without a new fixture and enum variant. Add a flat `ScenarioSpec` of closed enums as the composable, structured (AI-targetable) surface, and a deterministic `render()` that reuses the existing assembly seams (CbParams, cb_toml/_mux, build_mev_params, ElCl) verbatim — no new YAML/TOML emission. - The 13 named scenarios map to specs via `Scenario::to_spec()`, and `lower_reproduces_every_scenario` proves `render(spec) == args_file_in` byte-for-byte, so the composable path inherits the byte-golden coverage. - The per-scenario comment prose has no knob preimage, so it is a render-time parameter (named scenarios pass their verbatim comment; composed specs get a generated one), not a spec field. - Acceptance for the combinatorial space is offline property tests, not golden fixtures: a renderer-drift round-trip (`detect_enabled_features(render) == armed_features`, documented as drift-only — it is blind to shared typos, that is preflight's job), a composite fragment-order pin, and total-render over an enumeration. Mux + any injection feature is rejected loudly (no injection seam). - `sim scenario --spec <json>` (AI-drivable) or `--base <name> --set k=v,...` (deterministic keyword overlay) renders a composed config; `--show-spec` previews the resolved spec + what it arms. Validity by construction: output goes through the same seams the goldens pin, and a live preflight of a novel ws+prysm+timing compose passes helix config-parse. Design + the grill-driven cut list (no matrix verb, no per-spec expected-check table, no in-binary NL layer) in docs/composable-scenarios.md.
…tures proven on devnet)
Add teku, nimbus, lodestar as geth-paired ClientPair variants (the clients
axis was the weakest part of the composable spec — only 2 points, undercutting
Law 7). The CL is the axis that matters for CB behavior, so the new pairs vary
the CL against geth; the rpc_url naming (el-1-{el}-{cl}) is already parametric,
so this is an ElCl + ClientPair variant + serde name, no template changes.
Add spec::curated(): a handful of high-value composed specs frozen as
named+goldened regression anchors (tests/fixtures/curated-configs/), emitted by
`sim generate --curated`. Each is live-validated on a devnet before its golden
is trusted:
- cb-basic-{teku,nimbus,lodestar}: 14/0/0 each (full MEV pipeline, all hashes match)
- cb-ws-prysm: ws stream FIRES on prysm (30 headers, 1 startup-race fallback) —
refutes the Law-7 route-coupling concern by measurement
- cb-timing-extra-validation: both feature markers proven from CB logs
Curated goldens are bless-able via BLESS_CURATED=1 (only after a live run
confirms the config works).
Audit before merge found the scenario COUNT was stale everywhere it was
hard-coded ("six"/"nine"/"6 scenarios" in README, ARCH.md, the runbook) and
the new composable surface was undocumented in the routers.
- ARCH.md: add the genmodel/spec.rs module row (ScenarioSpec + render +
curated); drop the hard-coded "six scenarios" counts (reference Scenario::ALL).
- AGENTS.md: refresh the named-scenario list, add a "Composable scenarios"
paragraph (sim scenario, the 5-CL clients axis, curated), and update the
generate examples.
- README: honest "the named scenarios (Scenario::ALL); headline ones" intro +
the composable clients axis (all 5 CLs); local-kurtosis-e2e.md count dropped.
- Strengthen the documentation-discipline rule: the scenario/spec rows now list
every doc that must move together, and a new rule bans hard-coding a COUNT in
prose (cite the source of truth instead) — the exact drift this audit found.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.