Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 25 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,15 +81,27 @@ Against an already-running enclave (no observation window): `just verify-now <en

### Scenarios and config generation
`configs/generated/*.yml` are **gitignored build products** of the typed generator; the tracked truth is
`tests/fixtures/golden-configs/`. Nine scenarios (`Scenario::ALL`, `src/bin/sim/genmodel/scenario.rs`):
`cb-basic`, `cb-basic-nethermind-prysm` (Law 7 alt EL/CL pair), `cb-multiple-relays` (two helix instances,
divergent per-relay subsidies), `cb-skip-sigverify`, `cb-sigverify-diff` + `cb-sigverify-diff-control`
(a real ON/OFF differential), `cb-timing-games`, `cb-extra-validation`, `cb-mux` (256 validators split
across two relays).
`tests/fixtures/golden-configs/`. The named scenarios are `Scenario::ALL`
(`src/bin/sim/genmodel/scenario.rs`) — the frozen, byte-goldened regression set: `cb-basic`,
`cb-basic-nethermind-prysm` (Law 7 alt EL/CL pair), `cb-multiple-relays` (divergent per-relay subsidies),
`cb-min-bid`, `cb-signer`, `cb-skip-sigverify`, `cb-sigverify-diff` + `cb-sigverify-diff-control` (a real
ON/OFF differential), `cb-timing-games`, `cb-extra-validation`, `cb-ws-stream` + `cb-ws-stream-nokey`
(the ws stream + its negative control), `cb-mux` (256 validators split across two relays).

**Composable scenarios** (`src/bin/sim/genmodel/spec.rs`, [`docs/composable-scenarios.md`](docs/composable-scenarios.md)):
`ScenarioSpec` is a flat closed-enum surface that composes features freely and renders through the SAME seams
as the named scenarios (proven byte-identical for every named one via `Scenario::to_spec()` +
`lower_reproduces_every_scenario`). Use it for any combination that is not a frozen named scenario — e.g. the
ws stream on a specific client, or a client the named set does not cover. The `clients` axis covers all 5
mainstream CLs (`geth-lighthouse`, `nethermind-prysm`, `geth-teku`, `geth-nimbus`, `geth-lodestar`).
`spec::curated()` freezes a few high-value composed specs as goldens (`tests/fixtures/curated-configs/`).

```bash
sim generate # all nine -> configs/generated/ (= just generate-configs)
sim generate # the named scenarios -> configs/generated/ (= just generate-configs)
sim generate --curated # + the curated composable coverage configs
sim generate cb-mux --out-dir /tmp/x
sim scenario --base cb-basic --set clients=geth-teku,get_header=stream # compose (stdout)
sim scenario --spec spec.json # full ScenarioSpec as JSON (the AI-drivable surface)
sim generate --check # drift gate: nonzero if on-disk configs != what the generator emits
sim preflight configs/generated/cb-mux.yml # ~1s: parse the rendered config with the REAL helix image
sim checks --list [--json] # the check contract, machine-readable
Expand Down Expand Up @@ -257,7 +269,8 @@ you MUST update this file AND its companion doc IN THE SAME COMMIT:**
|---|---|
| a check's id, tier, verdict conditions, or data source | `docs/CHECKS.md` **and** `src/bin/sim/checks_catalog.rs` |
| a CLI flag, a `just` target, the launcher | this file + `README.md` |
| a scenario, the generator, the helix/CB blocks | `tests/fixtures/golden-configs/` (regenerate) + this file's scenario list |
| a scenario, the generator, the helix/CB blocks | `tests/fixtures/golden-configs/` (regenerate) + this file's scenario list + `README.md` + the `genmodel` rows in `docs/ARCH.md` |
| the `ScenarioSpec` surface, its knobs/clients, or `curated()` | `docs/composable-scenarios.md` + this file's "Composable scenarios" + `README.md` + `tests/fixtures/curated-configs/` (`BLESS_CURATED=1`, only after a live run) |
| a design law | `docs/DESIGN.md` |
| a ratified direction, a staged-plan status | `.agent/NORTH-STAR.md` |
| a live-devnet result or a new defect | `.agent/SWEEP-BACKLOG.md` (findings log) |
Expand All @@ -270,6 +283,11 @@ in `src/checks/cb_metrics.rs` while missing from BOTH `checks_catalog.rs` and `d
commit that added it (caught on the next read, now synced). Adding a check is a THREE-file
change. If a real registry ever lands in `src/checks`, derive the catalog from it and delete this note.

**Never hard-code a COUNT in prose** ("six scenarios", "all nine", "the two client pairs"). Counts drift the
moment a scenario or client is added and go silently wrong across every doc that repeated them (this happened:
"six"/"nine"/"6 scenarios" were stale in README, ARCH.md, and the runbook simultaneously). Reference the
source of truth instead — `Scenario::ALL`, the `ClientPair` variants, `curated()` — and describe, don't tally.

**New hard-won gotchas belong in the "Known traps" section above**, with the evidence that bought them.
A trap that lives only in a commit message will be paid for twice.

Expand Down
32 changes: 31 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,16 +64,46 @@ Kurtosis uses a default Commit-Boost config that can be overridden by inlining i
just generate-configs
```

Six scenarios are generated:
`sim generate` emits the named scenarios (the frozen, byte-goldened regression set — the full list is
`Scenario::ALL` in `src/bin/sim/genmodel/scenario.rs`). The headline ones:

| Config | What it tests |
|---|---|
| `cb-basic.yml` | Single relay (helix), default CB config |
| `cb-basic-nethermind-prysm.yml` | cb-basic on an alternate EL/CL pair (Law 7) |
| `cb-multiple-relays.yml` | Two helix relay instances, aggregated bidding |
| `cb-mux.yml` | Mux routing — 128 validators to helix-1, 128 to helix-2 |
| `cb-skip-sigverify.yml` | Fast path with BLS signature verification disabled |
| `cb-timing-games.yml` | Aggressive per-relay timing overrides for late bidding |
| `cb-extra-validation.yml` | Extra get_header validation via local EL RPC |
| `cb-ws-stream.yml` | getHeader over the websocket bid stream |

For any combination outside the frozen named set (a feature on a specific client, another CL, ...), compose
one with `sim scenario` — see [Composable scenarios](#composable-scenarios-sim-scenario) below.

### Composable scenarios (`sim scenario`)

The named scenarios above are frozen points. To compose features freely — e.g. the
websocket stream on the prysm client pair with timing games, a combination no named
scenario covers — use `sim scenario`, which renders a `ScenarioSpec` through the same
assembly seams the goldens pin (so a rendered config is valid by construction):

```bash
# Start from a named base and apply typed field overrides:
cargo run --bin sim -- scenario \
--base cb-basic --set get_header=stream,clients=nethermind-prysm,timing_games=true \
--show-spec --out configs/generated/cb-ws-prysm-tg.yml

# Or supply a full ScenarioSpec as JSON (the AI-drivable surface; unknown keys rejected):
echo '{"topology":"mux"}' | cargo run --bin sim -- scenario --spec /dev/stdin
```

Overridable knobs: `clients` (geth-lighthouse | nethermind-prysm | geth-teku | geth-nimbus | geth-lodestar —
all 5 mainstream CLs), `topology`
(single | two-relays | divergent-relays | mux), `get_header` (http | stream | stream-nokey),
`sigverify` (on | skip | skip-poisoned | poisoned-control), `min_bid` (none | `<eth>`), and the
booleans `timing_games` / `extra_validation` / `signer`. `--show-spec` previews the resolved spec
and the features it arms. Design + rationale: [`docs/composable-scenarios.md`](docs/composable-scenarios.md).

## Quick start

Expand Down
7 changes: 4 additions & 3 deletions docs/ARCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,9 @@ different report types because `sim` runs before/around a devnet, not against a
| `main.rs` | Dispatch to the three verbs; `tracing` init; nonzero exits on error. |
| `generate.rs` | IO boundary for `sim generate`: `.env` image overrides, atomic assemble-then-write, and `--check` drift gate (`generate::check`). Pure assembly lives in `genmodel`. |
| `genmodel/mod.rs` | Golden-fixture harness (`golden`, `assert_matches_golden`, `extract_block_scalar`) — the byte-identity oracle for the verbatim config port. |
| `genmodel/scenario.rs` | `Scenario` enum (six scenarios) + `Images` map; `args_file_in()` joins the static fragments + helix const + CB block into a full args-file; `build_mev_params`. |
| `genmodel/helix.rs` | `HELIX_RELAY_CONFIG` — the helix YAML block, byte-identical across all six scenarios (const). |
| `genmodel/scenario.rs` | `Scenario` enum (the named scenarios, `Scenario::ALL`) + `Images` map; `args_file_in()` joins the static fragments + helix const + CB block into a full args-file; `build_mev_params`. `to_spec()` maps each named scenario to a `ScenarioSpec`. |
| `genmodel/spec.rs` | `ScenarioSpec` — the flat, composable, structured (AI-targetable) scenario surface (closed-enum knobs: clients / topology / get_header / sigverify / min_bid / timing_games / extra_validation / signer). `render()` reuses the same assembly seams as `args_file_in` (proven byte-identical for every named scenario), so any composition renders a valid config. `curated()` = high-value composed specs frozen as goldens. Drives `sim scenario` + `sim generate --curated`. See [`composable-scenarios.md`](composable-scenarios.md). |
| `genmodel/helix.rs` | `HELIX_RELAY_CONFIG` — the helix YAML block, byte-identical across all named scenarios (const). |
| `genmodel/cb.rs` | The CB TOML block: `cb_toml(CbParams)` + `cb_toml_mux(node0, node1)` — verbatim port of the Python builders, generate-time knobs injected by string building. |
| `render.rs` | The **compatibility contract** with the fork: `extract_config_blocks` (pull the two `|` scalars), `substitute_runtime_vars` (strip the `{{ range }}` loop, fill `{{ .VAR }}` dummies), `default_dummies`. Pure. |
| `preflight.rs` | `sim preflight`: render + run the real helix image + `classify_helix_probe` (pure, 3-valued). |
Expand All @@ -130,7 +131,7 @@ different report types because `sim` runs before/around a devnet, not against a
This is the load-bearing seam. `sim generate` emits a Kurtosis args-file whose `mev_params` carries
**two `|` block scalars** that the forked ethereum-package parses and fills at launch:

- `helix_relay_config: |` — the helix relay's YAML config (byte-identical across all six scenarios).
- `helix_relay_config: |` — the helix relay's YAML config (byte-identical across all named scenarios).
- `commit_boost_config: |` — the commit-boost sidecar's TOML config (varies ≤ ~7 lines per scenario).

Both blocks are **opaque scalars to YAML** and contain unrendered **Go-template holes** that only the
Expand Down
7 changes: 7 additions & 0 deletions docs/DEVELOPING.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,13 @@ A scenario is a typed devnet configuration that assembles into a Kurtosis args-f
fixtures. The config↔fork coupling (the two `|` block scalars, the runtime template holes) is explained in
[`docs/ARCH.md`](ARCH.md) §4 — read it before touching the block bodies.

> **Composing instead of authoring.** For a one-off combination of *existing* features (e.g. the ws stream on
> the prysm pair with timing games), you do not need a new named scenario — use `sim scenario --base/--set` or
> `--spec <json>` (`genmodel/spec.rs`, [`docs/composable-scenarios.md`](composable-scenarios.md)). Add a NEW
> named scenario + golden below only when a combination is worth freezing as a regression anchor, or when a
> feature needs a config knob no `ScenarioSpec` field exposes yet (add the field, keeping
> `lower_reproduces_every_scenario` green).

### Steps

1. **Add the variant** to the `Scenario` enum in
Expand Down
Loading
Loading