Feat/sim harness - #2
Merged
Merged
Conversation
runbook
Reconcile the embedded helix config with current helix-relay:main:
- drop the removed `network_config: !Custom {...}` block (the relay now
fetches
chain spec + genesis from the beacon node at startup)
- migrate `cores` to the 10-field CoresConfig (drop sub_workers; add
decoder,
simulator, top_bid, data_gatherer, block_merging, housekeeper)
Bump the ethereum-package fork to the helix wait-for-genesis fix. Add
docs/local-kurtosis-e2e.md, a full runbook (build CB image -> deploy
devnet ->
run cb-basic -> verify) with every gotcha (kurtosis 1.18.1 pin, the 3
helix fixes,
pin-vs-latest coupling).
Introduce the `sim` Rust binary: the agent-drivable launch->triage->diagnose
loop the release flow never had. Library-first (extract src/lib.rs so cb-verify,
cb-orchestrator, and sim share one core, killing the duplication at the source).
- `sim preflight <args-file>`: render + have the REAL relay/CB image parse the
config in ~1s, before any ~10-min devnet spend. 3-valued verdict
(Pass | Fail{field} | Inconclusive{reason}) so config-schema drift is caught
as a labeled failure, never a masked runtime panic minutes into launch. Env /
pre-genesis / pull issues classify Inconclusive, not a false Fail.
- `sim triage <enclave>`: attach to a broken enclave, dump each non-RUNNING
service's ROOT panic via a docker-logs fallback that pierces kurtosis's
grpc/masking. Structured TriageReport JSON.
- Gate wiring in run-and-verify.sh: preflight blocks launch on Fail; triage
auto-fires on any launch failure and surfaces its JSON. Known digest-window
limitation documented at the site (deferred to `sim run` in P2).
Proven end-to-end vs ghcr.io/gattaca-com/helix-relay:main: valid cb-basic.yml ->
pass/exit0/~8.5s; renamed `hostname` field -> Fail{hostname}/exit1/~0.3s.
77 tests green (pure cores TDD'd on real + held-out fixtures; IO wirings are
Docker/kurtosis smoke checks per Law 4).
Also lands docs/NORTH-STAR.md (ratified direction: full-Rust sim, owned helix
mirror, dogfood one fork) and the P1 plan.
Mechanical clippy fixes across the pre-sim code (collapsible_if -> let-chains, while_let_loop -> for/by_ref, manual_strip -> strip_prefix, unnecessary_unwrap, useless_format/vec, needless_borrow, expect_fun_call). No behavior change; 77 tests still green. Two justified #[allow]s (with comments): too_many_arguments on the enclave pipeline launcher (an options struct would read worse), and dead_code on two relay data-API Deserialize targets whose unread fields document the wire schema. `cargo clippy --all-targets -- -D warnings` now passes clean, so every subsequent commit builds under a real warning gate.
…grill)
Three adversarial reviews + a direct diff killed the typed-serde-mirror plan:
the "6 duplicated templates" premise is false (helix is byte-identical across
all 6 scenarios; CB varies <=7 lines), typed helix gains no guard (types aren't
importable; preflight stays the only check), and the serde mechanism is fragile
(sentinel collisions: 2000 vs late_in_slot_time_ms, db_name==user=="helix").
A code trace also refuted "just delete templating" — the ethereum-package fills
the {{ }} holes from runtime service-discovery values at launch.
New plan: port the (already-DRY) Python templates verbatim into Rust const
strings (holes stay literal — no serialization, no sentinel hazard); type only
the assembly (Scenario enum + one Images map). Byte-identity to golden is the
oracle. Fixes the live-wrong commit-boost/pbs image default along the way.
…sk 0) The 6 Python-generated configs that produced the green e2e run, snapshotted with the baked-default images (commit-boost/commit-boost, public helix) so the fixtures are hermetic — not tied to the box's .env. `genmodel::assert_matches_golden` is the acceptance oracle for the verbatim port to come: byte-identity, naming the first differing line on mismatch. A guard test asserts the pbs image bug is absent from the fixtures.
Ports scripts/generate_kurtosis_configs.py into the sim binary. The config
bodies are verbatim const/string templates (helix keeps its ~40 lines of
binary-verified drift comments; the {{ }} runtime holes stay literal — the
ethereum-package fills them at launch). Typing lives only at the assembly
layer: a Scenario enum + one Images map with the CORRECT commit-boost/commit-boost
default (fixing the live-wrong commit-boost/pbs Python default).
All 6 scenarios reproduce their golden fixtures byte-for-byte (the acceptance
oracle); cb-basic still passes sim preflight against the real helix image.
Review fixes folded in (adversarial review before commit):
- load_pubkeys returns Result; run assembles ALL bodies before writing any, so a
missing/malformed mux keys file fails cleanly with nothing written (matches the
Python's pre-write all-or-nothing), instead of panicking mid-generation with 5
stale files left behind.
- run_in(keys_dir, env_path) injectable core for hermetic negative-path tests.
- Scenario::ALL reordered to the true Python emission order; misleading
generate_all_matches_goldens renamed to reflect it checks IO faithfulness (the
hermetic golden match lives in scenario.rs on default images).
43 sim tests green (incl. missing-keys → clean-error + atomicity); clippy -D clean.
…e one source Deletes scripts/generate_kurtosis_configs.py (Law 1's root smell — it reverse-engineered helix's serde layout from binary panics) and the stale configs/example-kurtosis-config.yml hand-copy. `just generate-configs` now runs `sim generate`; the tracked cb-basic.yml regenerates byte-identically. Doc repointing: README image-default table fixed to the baked defaults (commit-boost/commit-boost, public helix — no more commit-boost/pbs), the config-gen section + repo-layout tree point at `sim generate`, and the runbook's helix-drift fix instruction now targets src/bin/sim/genmodel/helix.rs's HELIX_RELAY_CONFIG const instead of the deleted Python template.
NORTH-STAR staged-plan P2 now reflects what actually shipped (verbatim port + assembly typing, not cb_common mirrors) with a note that the grill outcome tensions with Law 1's "built from cb_common structs" premise — J's call to revise the Law. P2 plan doc stamped LANDED with the commit trail.
…call to J Probed commit-boost/commit-boost:kurtosis: CB parses eagerly + errors structured (chain-file-not-found at types.rs:442 is the clean post-deserialize PASS marker, no spec mount needed). BUT [pbs] is #[serde(flatten)] + const-defaults, so a renamed pbs field is silently defaulted and reaches PASS — a CB preflight would false-pass the likeliest drift class. Not shipping an instrument that can lie (pilot-breaks-the-instrument); deferred to J with the honest-Inconclusive vs partial-preflight tradeoff written up.
…enerate configs/generated/cb-basic.yml is tracked (render.rs's fixture + what preflight validates); a new test asserts sim generate reproduces it, so a hand-edit or a stale regen fails CI instead of silently diverging — the staleness class that rotted the old example config.
…ching Adds decision-logic unit tests for the two Tier-1 checks that had zero: - chain_health: inverted-range → Fail; single-slot window → Skip (not a false pass on zero data) - payload_matching: empty relay set → Skip (not a false pass on zero comparisons) Only the pre-fetch guard branches are unit-testable today — both checks inline their real decision logic inside async fetch fns (no pure classifier seam, unlike cb_metrics.rs). The deeper boundaries stay uncovered pending a classifier extraction (flagged for P3). No production logic changed.
… approach Consolidates the three code-verified false-greens (mux_routing pass-gate, relay_pipeline + payload_matching first-wins-union-by-slot) with their shared structural root cause: decision logic inlined in async fetch fns → no test seam → bugs hide. Fix = extract pure classifiers (the healthy cb_metrics.rs pattern), which serves Law 3 (no false greens) and Law 4 (testable verdict math) at once. Lays out the per-check judgment calls (warn-vs-fail, tier, CB-debug-logging) for J. Proposal only — verdict-changing, not started autonomously.
`sim generate --check` regenerates in-memory and verifies the on-disk configs already match, exiting nonzero (naming the drifted files) without writing — so CI or an agent can assert the checked-in configs are current in one command, instead of only catching drift via `cargo test`. Shares the assembly path with `run` so both fail identically on a bad mux keys file.
Deleting configs/example-kurtosis-config.yml in the P2 retire commit left integration.yml pointing at a nonexistent file (my Task 2 grep missed .github/). Fix: generate cb-basic via `sim generate` and launch that, pinning the public commit-boost image the old example used (the nightly builds no CB image). Also align .env.example's documented defaults with Images::default and drop its reference to the retired Python generator. Flagged in-workflow: confirm the public pbs:latest tag is SSZ-current or switch to building the image.
This session's subagent edits (P1 sim bins, the clippy cleanup, P2 genmodel, the Law 4 tests) weren't run through rustfmt, so `cargo fmt --check` (a CI job) was failing. Pure formatting — rustfmt is AST-based and all 99 tests still pass; byte-identity of `sim generate` output is unaffected (string literals untouched). The branch now passes check + test + clippy + fmt-check.
…ings mux → WARN when unverifiable (debug logging already on in generated config); payload → detect per-(relay,slot) conflict, verdict stays WARN; best-bid → new arm comparing per-relay bid values from CB logs (keep delivered-count as coverage); CB-preflight → keep honest Inconclusive.
… (P3)
The false-green: mux.routing PASSed whenever there were no misrouting violations,
even when ZERO routing decisions were actually checked (CB "using mux config"
DEBUG events absent) — reporting "All N verified" where N counted raw log lines.
Fix (J's call): extract a pure `classify_mux_routing` (the Law 4 test seam) and
count `routing_decisions_verified` = events with a known pubkey AND a mux_id. If
that's 0, WARN ("need CB debug logs") instead of PASS; PASS now counts verified
decisions, not log lines. Debug logging is already on in the generated cb-mux
config, so real runs stay green — only the unverifiable case flips to WARN.
4 decision tests added (pass / warn-unverifiable / warn-no-events / fail-misroute).
The false-green: `by_slot.entry(slot).or_insert(hash)` was first-wins union, so when two relays reported DIFFERENT block_hash for one slot (relay equivocation — the thing the cross-check exists to catch) all but the first were dropped before the on-chain compare, and the verdict was order-dependent. Fix (J's call): collect every (relay, hash) per slot; extract a pure generic `classify_payload_matches` (Law 4 seam); WARN on a cross-relay conflict OR when no relay hash matches chain. `missed` stays informational (doesn't downgrade), matching prior behavior. 5 decision tests (u64 stands in for B256).
… bid values (P3) The false-green: check_payloads_delivered_multi unions by slot and counts distinct delivered slots, so one delivering relay scored identically to genuine two-relay aggregation — the "aggregated bidding" the multi-relay scenario exists to test was never actually checked. Fix (J's call): keep the delivered-count as a COVERAGE check and ADD a real best-bid arm. check_best_bid_selection gathers each relay's best bid per slot (max of its builder-blocks-received) + the delivered value, then classify_best_bid (a pure Law 4 seam) asserts: WARN if no slot had >=2 relays competing (aggregation never exercised — the anti-false-green), WARN if a competitive slot delivered LESS than the best available bid, else PASS. 4 decision tests (u64 stands in for U256). Data-semantics assumption to confirm in review: a relay's offered bid = max of its builder_blocks_received for the slot; delivered value is comparable to bid value.
…lay' A single relay reporting two hashes for one slot is also a conflict; the message now says 'N distinct block hashes reported' rather than asserting cross-relay disagreement. Verdict unchanged (WARN).
The adversarial review found check_best_bid_selection's data source wrong: get_builder_blocks_received returns EVERY builder submission a relay received (including bids that failed simulation and were never offered to the proposer), so max(builder_blocks) overstates the relay's offered bid and would false-alarm "value left on the table" on correct runs. The ~11-slot sampling also made the "no competition" WARN the default even when aggregation works. The pure classify_best_bid verdict logic was sound; only the IO source was wrong. Correct design (P3 follow-up): source per-relay bids from the CB "received new header" log events (relay_id + slot + value_eth, already parsed by parse_cb_log_line, full-coverage not sampled), parse value_eth decimal->wei for Ord, and compare the delivered value to the max OFFERED bid per slot. That needs CB-log context (enclave + cb_service_names) plumbed into the check — a focused, separately-reviewed piece rather than a rushed trust-core patch. check_payloads_delivered_multi stays as the coverage check. mux.routing and payload_hash_match fixes are unaffected.
…ng CB-log source Records the adversarial-review outcome: WARN is non-fatal (won't break CI), the best-bid data-source problem + correct follow-up design, and the lenient gaps left within J's ratified scope.
…iew fix) Re-adds the aggregated-bidding check with the CORRECT data source the review demanded. Per-relay OFFERED bids now come from CB's own "received new header" getHeader log events (relay_id + slot + value_eth, full-coverage, already parsed by parse_cb_log_line) — what CB actually compared — instead of the relay data API's builder_blocks_received (which includes bids that failed sim and were never offered, and was slot-sampled). Fixes both review findings: no false "value left on the table" alarms, and no sampling-induced "no competition" default. value_eth is parsed decimal->wei WITHOUT float (exact); bids and delivered are compared in GWEI so sub-gwei cross-source rounding can't spuriously warn. Single- relay runs SKIP. The pure classify_best_bid seam is unchanged. New module src/checks/best_bid.rs; wired in main.rs alongside the mux check (needs CB-log context). 4 tests incl. exact value_eth parsing.
…wei (P3 review 2)
Second review of best_bid v2 found the crux sound ("received new header" IS the
per-relay offered bid) but caught a reintroduced Law-3 false-green: competitive
slots with NO delivered payload (out of window / missed) counted toward the PASS
"verified across N" claim. Now the verdict is based on VERIFIED slots (competitive
AND delivered present); if none are verified → WARN, never PASS. Also: window-filter
the bid harvest to [start,end]; drop the gwei rounding (both sources are exact wei,
so it only added a boundary artifact) and compare wei directly; soften the
suboptimal message (dropped the unfounded "misrouting" claim). +1 guard test.
Closes the from-scratch gap: cb-testing had no recipe to build the CB image (it lives in the sibling commit-boost-client repo) or pre-pull the public images. - build-cb-image [tag] [cb_dir] — delegates to ../commit-boost-client `just build-all` - pull-images — pre-pull helix/reth-rbuilder/lighthouse (public; helix isn't built) - e2e [config] — generate-configs + pull-images + testnet, one command README quickstart leads with the build-cb-image (once) -> e2e path.
Two robustness fixes surfaced by the live-devnet validation: - Pre-build cb-verify (release) BEFORE `kurtosis run` and invoke the built binary, instead of `cargo run --release` mid-devnet — that compile is a multi-GB spike while 10 services are live. Keeps it off the critical path. - A host-memory advisory before launch (non-blocking; LOW_MEM_ABORT=1 to abort). NOTE: this is a general host-resource guard, NOT the cause of the relay OOMs seen on 2026-07-31 — those were per-container cgroup OOMs (CONSTRAINT_MEMCG) at the relays' own RELAY_MAX_MEMORY cap, addressed separately in the fork.
Helix is industry-predominant; the flashbots mev-boost-relay is a memory hog
(leaked ~825MB/min under spamoor). So the multi-relay scenarios (cb-multiple-
relays, cb-timing-games, cb-mux) now run TWO helix instances instead of
helix+flashbots. flashbots stays as the block BUILDER (reth-rbuilder).
- scenario.rs relays() -> ["helix","helix"] for the three multi-relay scenarios;
fixed the now-inaccurate comment prose baked into the configs.
- cb.rs mux labels node_1_to_flashbots/mux_flashbots -> node_1_to_helix/mux_helix_1
(cosmetic; routing is positional {{ index .Relays 0/1 }}).
- Regenerated the 3 multi-relay goldens (now the intended 2-helix output, not the
old Python baseline — mod.rs provenance updated).
- Bumps the ethereum-package submodule to the N-instance launcher (fbe3141).
Validated on a live devnet: helix-relay-2 + helix-relay-3 both survive, and every
multi-relay check PASSes on real data — best_bid 33 competitive/33 verified,
payload_hash 33 matched/0 conflicts, delivered across 2 relays. (Residual overall
FAIL is transient warmup 5xx: 1 get_header + 8 submit, down from 42+123 when the
flashbots relay was OOM-dying.)
…a staleness Beefs up documentation after the P1/P2/P3 + 2-helix work, from a six-lens sweep: NEW cross-cutting docs (the biggest gaps): - docs/CHECKS.md — authoritative per-check catalog: tier, pass/warn/fail contract, data source, and the load-bearing verdict rule (exit keys ONLY on tier-1 FAIL; WARN/SKIP non-fatal, so a consumer must parse JSON result, not just exit code). - docs/ARCH.md — how it all fits: sim generate → run-and-verify → cb-verify (discovery→probes→checks→report) → fork; module map; key decisions. - docs/fork-delta.md — the ethereum-package divergence from upstream (mev_resolver component model, helix wait-for-genesis + N-instance + 8GB cap, rebase notes). - docs/SWEEP-BACKLOG.md — the prioritized backlog (bugs/docs/tests/perf/refactor/ features); docs/plans/INDEX.md — landed-vs-live plan classification. FIXES to now-stale docs: - README + .env.example: multi-relay = two helix instances (flashbots relay dropped, builder only); added relay.best_bid + sim verb contract + --skip-finalization-check + a pointer to CHECKS.md. - Re-stamped P1/P2/P3 plan statuses (all shipped; read "uncommitted/backed-out").
cb-orchestrator (just test-all/test-one) invoked the cb-verify binary with --cb-config, which cb-verify's clap doesn't define, so clap rejected it and the batch runner failed at every check invocation. Surfaced by the doc sweep.
…ercount C1 (CRITICAL): when all relays are unreachable at check time (mid-run OOM death — the exact scenario this repo exists to catch), run_relay_checks emitted SKIP for the tier-1 relay.payloads_delivered_multi, and exit_code treats a tier-1 SKIP as pass → the run exits 0/PASS with the MEV pipeline unverified. Extracted a pure `all_relays_dead_results` that FAILs the tier-1 delivery check (tier-2/3 stay SKIP, already gated). Unit-tested. H3: check_mev_delivery_rate broke on the first relay that answered the data API, so under mux (each relay holds only its half of deliveries) it undercounted and spuriously WARNed. Now unions delivered payloads across ALL relays (like check_payloads_delivered_multi); SKIP only if NO relay answered.
probe_all returns Vec<(label, error)>, not Vec<label>. The previous commit shipped a test that did not compile: I grepped for 'test result:' lines and treated their ABSENCE as silence rather than as failure, so a broken build read as green. Same class of mistake the harness itself keeps finding - a check that cannot distinguish 'no signal' from 'bad signal'. Test now asserts the labels AND that the death reason is carried rather than discarded. Gates re-run with explicit exit codes (test/clippy/fmt all 0): lib 160, sim 72, cb-verify 10.
… produced a false PASS I built on
An independent audit of CB's metric surface caught both. They matter because
neither failed loudly; both reported green forever.
1. cb_relay_v2_unsupported never fired. CB's PBS registry is
Registry::new_custom(Some("cb_pbs")), and this counter is REGISTERED as
`pbs_submit_block_v2_unsupported_total`, so the EXPOSED name carries a
doubled prefix: `cb_pbs_pbs_submit_block_v2_unsupported_total`. Every sibling
is registered bare (`relay_status_code_total` -> `cb_pbs_relay_status_code_total`),
which is exactly why the odd one out slipped through. We matched the
registered name, so the check PASSed by construction.
RETRACTION: that false PASS was my only evidence that enabling helix's
GetPayloadV2 route fixed the nethermind+prysm failure. It was structural, not
evidence. submit_blinded_block failed identically before and after (26 vs 25
rejected), so the fix's EFFECT is now unverified - re-test before claiming it.
Adding the route is still correct on its own merits; its consequence is not
established.
2. cb_v2_fallback was permanently green. It read
`cb_pbs_submit_block_v2_fallback_to_v1_total`; commit-boost registers no
`*fallback*` metric anywhere. "Counter absent" was treated as "zero fallbacks
== PASS", so it could never fail - and it had also asserted "relays support
v2" on a run where the relay was 404ing v2. Now SKIPs, naming itself inert
and pointing at the check that owns v2 support.
Method rule: verify metric names against a REAL scrape, not against the
registration constant in CB source - the registry prefix is applied at gather
time. Catalog + backlog synced; tests re-pointed at the exposed names.
An independent adversarial review killed the key layout (secrets/ is chmod 0600 -R, so uid 10001 cannot traverse it and the signer silently loads ZERO keys; use teku-keys/teku-secrets, which web3signer already proves readable) and killed launch Option A (the CB config artifact is not rendered until downstream in main.star). It also corrected an error in my own research: a MISSING [[modules]] bails loudly, not silently. Plus: pbs.with_signer is dead code in the shipped binary, so 'PBS uses the signer' is not an available escalation; the assertion ladder was mostly vanity (/status is an unconditional 200 and there are two of them; loaded_consensus is log-only and ANSI-mangled) and collapses to a JWT-authed get_pubkeys COUNT assertion; and two afternoon-burning traps (negative controls rate-limit our own IP for 300s; a commit-boost-* service name adds a 200k-line log fetch per check).
…(+ flatten-trap canary) First scenario from the CB config-surface audit: 5 of 21 PbsConfig fields were exercised by any scenario. min_bid_eth ranked highest because a wrong value degrades SILENTLY (too high => every bid dropped, which reads as "no MEV") and because it is observable. The audit's own premise was stale, though, and the scenario only works once corrected: it assumed ~0.05 ETH bids from old fixtures, but our scenarios set mev_builder_subsidy: 1, so real bids land near 1.04 ETH (measured: 1.0439 / 2.0439 in the divergent run) - and CB validates min_bid_wei < 1 ETH, so NO legal floor could ever reject a subsidized bid. cb-min-bid therefore runs with subsidy 0 (bids ~0.04 ETH of spamoor MEV) against a 0.5 ETH floor. feature.min_bid FAILs iff an auction winner's value is BELOW the floor - the definitive falsifier, since that is only possible if the floor was not applied. That makes this the canary for CB's silent-flatten trap: [pbs] cannot carry deny_unknown_fields (it must flatten PbsConfig), so a renamed/misspelled key there is ignored rather than rejected. Nothing-rejected is a WARN, not a red: "key ignored" and "every bid cleared the floor" look identical. 5 tests; the catalog's feature-invariant test caught the new id, as designed. lib 157 -> 162; 10 scenarios; catalog + CHECKS.md synced.
Checked on a live enclave before writing any starlark: secrets/ is mode 600 root:root (no execute bit, so uid 10001 cannot traverse it) while teku-secrets is 755 and teku-keys is 777. The originally-planned lighthouse layout would have produced a healthy signer holding ZERO keys. Kurtosis does not chown on mount, which is also why six other launchers force User(uid=0).
Pre-fix runs (x3): submit_blinded_block 222 v1 (200), 0 v2 (202). Post-fix cb-basic: 0 v1 (200), 222 v2 (202). A complete flip, on LIGHTHOUSE - so enabling helix's GetPayloadV2 changed the relay-side path for every scenario, not only the prysm one. This is counter evidence, not the structurally-broken check that produced the earlier false PASS. Also records a sweep caveat: run-and-verify.sh rebuilds cb-verify per scenario, so a sweep spanning code changes mixes binary versions - cb-basic's v2 checks came from the pre-fix build.
…r CL lighthouse over the relay's v2 route gets 202 Accepted (222 deliveries, PASS); nethermind+prysm gets 4xx on all 25. So the variable is what CB forwards when the request ORIGINATES from prysm (which calls CB's own v2 endpoint at 256ms into the slot, ruling out timing), not the route or relay capability. Next: read helix's rejection reason for a v2 submission and diff the content-type/body between the lighthouse-origin and prysm-origin paths.
Adds an OPT-IN CbParams.signer that appends [signer] + [signer.local.loader] +
[[modules]] to the CB config, plus a cb-signer scenario. Built to the adversarial
grill's verdict, not to my original plan - it killed two of my choices:
- **teku keystores, not lighthouse.** Verified live on an enclave: the package's
secrets/ dir is mode 600 root:root with NO execute bit, so CB's uid 10001
cannot traverse it and the signer would start healthy holding ZERO keys (the
loader is filter_map + warn!). teku-secrets is 755 and teku-keys 777 - the
same pair the package's own web3signer launcher already relies on.
- **Key paths stay placeholders.** They are per-participant
(node-<idx>-keystores/...) and the config template only carries
.Network/.Port/.Relays/.Timestamp, so they cannot be templated. CB's
CB_SIGNER_LOADER_{KEYS,SECRETS}_DIR env vars override the TOML at runtime.
- **Opt-in so the nine existing goldens stay byte-identical**, and appended
AFTER [logs.file] because interleaving top-level tables is invalid TOML once
[[relays]] has opened an array-of-tables (asserted by test).
- **A [[modules]] entry is mandatory**: with none CB bails loudly, with an empty
list it exits 0 SILENTLY. Every field is required, and signing_id must be
non-zero (asserted).
Generated TOML verified to parse and to match CB's schema shape. 4 tests;
sim 72 -> 76. Launcher/starlark half still to come.
…onfig Completes the config-gen half: the cb-signer scenario now also sets mev_params.commit_boost_signer so the fork launches the signer container. All 11 configs regenerate clean; the other 10 goldens are untouched.
…OUNT The third piece of the signer North Star: what actually falsifies. Built to the grill's collapsed ladder, deliberately NOT asserting the obvious: GET /status is `Ok(StatusCode::OK)` with no logic (200 with zero keys loaded), and the metrics server exposes a SECOND unconditional /status, so probing the wrong port is an even emptier green. The startup log's loaded_consensus=N is log-only (the signer registers exactly one metric, signer_status_code_total, with no key-count gauge) and ANSI-colored, so the field is not even a contiguous substring. Instead: a JWT-authenticated GET /signer/v1/get_pubkeys with a COUNT assertion. One HTTP call subsumes liveness, module registration, JWT auth AND key loading - and it is by construction the assertion that fails if the keystore mount is unreadable, which is this feature's most likely failure (CB's loader is filter_map + warn!, so a permissions problem yields a healthy signer holding nothing). Zero keys therefore FAILs and names the teku fix in the detail. The JWT is hand-rolled HS256 rather than pulling `jsonwebtoken`, because the claims are what must be exactly right: `route` binds to the exact request path, and `payload_hash` must be NULL on a bodyless request (CB enforces both directions). All of it is a pure, deterministic function with `now` injected - 5 tests decode the claims back and assert route binding, null payload_hash, expiry and secret-sensitivity. Also: the negative control classifies 401 PASS / 200 FAIL (auth not enforced) / 429 WARN naming the self-poisoning trap - CB rate-limits a source IP for 300s after 3 failures, and every harness request shares one NAT address, so negatives must run last. Discovery gains signer_urls on a SEPARATE `cb-signer-*` pattern, asserted not to leak into cb_service_names (three checks iterate that list shelling a 200k-line log fetch each). 11 tests; lib 162 -> 173.
The check module existed but nothing called it, so cb-signer would have run and asserted nothing - a scenario that cannot fail is the exact defect class this harness keeps finding. Now: for each discovered cb-signer-* service, mint a module JWT and assert the get_pubkeys COUNT against the devnet's active validator set. A non-200 or an unreachable signer FAILs at tier 1 rather than being silently absent. Catalog + CHECKS.md synced (the catalog's feature-invariant test caught the new id again, as designed).
…e trap to CLAUDE.md
…audit Audited every metric name the checks read against CB's registered names plus its registry prefixes. Result: the two bad ones are already handled and the rest are correct - a clean negative result worth banking rather than re-deriving. The rule that produced the bug: Registry::new_custom(Some(..)) prefixes every metric at gather time, and most PBS metrics are registered bare - but two carry their own prefix and end up DOUBLED (cb_pbs_pbs_submit_block_v2_unsupported_total, and cb_signer_signer_status_code_total, which we do not read yet but will if a signer metrics check is added). The test pins them so an 'obvious tidy-up' of the doubled prefix breaks loudly instead of silently disabling a check. It deliberately cannot detect a CB-side rename - only a real scrape can, which is the standing rule: verify metric names against a scrape, never against the source constant.
…lse red my own feature caused cb-multiple-relays FAILED the sweep on submit_blinded_block 186/626 5xx (29.7%) while the pipeline was flawless: 65/65 payloads delivered across 2 relays, 100% MEV rate, 65/65 payload hashes matched, best_bid verified over 65 competitive slots, 0 missed slots. The per-relay split shows why: mev_relay_0 (subsidy 1, LOSES every auction): 202x1, 4xx x219, 5xx x185 mev_relay_1 (subsidy 2, WINS every auction): 202x219, 4xx x1, 5xx x1 beacon side: 202x220 <- CB served the CL every time CB asks EVERY configured relay for the payload; only the auction winner has it, so the losers error by construction. My own divergent-bid change made relay_0 lose 100% of auctions (it previously split wins), pushing the relay-side rate from 18% to 29.7% and across the 25% line. The check was reading the wrong side of the proxy. Now judged on the beacon side - did the proposer actually get its payload? - with relay-side codes kept as diagnostic context. This is not a loosening: on the nethermind+prysm run the beacon side is 26x 5xx (CB returning 502 to the CL) and still FAILs. Falls back to relay-side logic when no beacon samples exist. 4 tests, both fixtures taken from REAL runs with opposite outcomes, which is what makes the discriminator trustworthy rather than a guess. CHECKS.md + catalog carry the evidence.
…d, JWT authenticated signer.pubkeys PASS on a run that also PASSes overall. First time the Commit-Boost signer has been exercised on a Kurtosis devnet. Cost after the grill: one devnet run on a one-line type error; every trap the grill named was avoided before it fired. Caveat recorded honestly: the container's own logs were not captured (my capture grep matched the enclave name, and the script tore down despite --keep), so loaded_consensus was never read as corroboration.
…he run Tier-1 WARN is non-fatal, so a scenario whose whole purpose is exercising one feature could report "NOT asserting the feature ran" and still exit 0. A sweep counted that as a win. Law 7 caught a real upstream bug today; Law 3 could not, because its failure mode was green. Separate the two things that both landed on WARN: - armed but unmeasured: the differential was set up and produced no signal. That is a failure to measure, not an anomaly. Now flagged `inconclusive`, and `--require-feature-proof` makes a tier-1 inconclusive check exit 1. - structurally unconfirmable, or a genuine anomaly: skip_sigverify unpoisoned is a negative codepath that emits nothing when it fires, and relay equivocation is a real observation. Both stay plain WARN. The flag defaults OFF, so the exit contract in docs/CHECKS.md is unchanged for existing callers. Turning it on is the sweep's call. When it does go on, cb-min-bid goes red - correctly: its zero rejections come from the builder subsidy lifting every bid over the floor, so it proves nothing today. Tests cover both sides of the seam, including the negative case pinning that unpoisoned skip_sigverify is NOT marked. Without it, "tidying" that site would turn every scenario carrying skip_sigverify permanently red.
All five "highest leverage" items had been fixed after the section was written, and it was never revised. A fresh session read it, relayed it as the plan, and started reworking C1 before checking the code. That is a false GREEN in the planning layer, the same shape as the harness bugs the list describes. Mark each closed with the evidence that falsifies it (commit, test name, grep), and record the rule it bought: a do-first entry must carry the evidence that would falsify it, and you verify against the code before working it off.
…riminator CB main merged the ws getHeader stream WITH an HTTP fallback, so a broken stream passes every MEV check silently. These criteria are what catch that: - Feature::WsHeaderStream: armed by `get_header = "stream"` (Feature gains config_value(); this is a string knob, not a boolean), proof marker "received new header from ws stream", tier-1, inconclusive when armed but silent - so --require-feature-proof reds a fallback-only run. No separate relay-side check: a streamed header in CB's log can only exist because the relay delivered it, so one marker proves the full path. - classify_ws_fallback: thresholds MEASURED, not guessed (two 220-slot runs on CB e622a5e + helix :main each showed exactly one startup-registration-race fallback): 0 -> PASS, 1-with-stream -> PASS, flapping -> WARN, never-served -> WARN annotative only (the marker check owns the inconclusive flag; tests pin no double-flagging). - Marker strings pinned by test against CB main's actual log lines - reworded logs upstream fail here first. Scenarios: cb-ws-stream (generated, golden-pinned; live run PASSED under --require-feature-proof: 220 markers, 1 fallback) and cb-ws-stream-nokey, the NEGATIVE CONTROL - no api key, helix refuses every handshake, expected to FAIL under --require-feature-proof; that failure is the proof the criteria discriminate. Not part of the green sweep. Also: HeaderStream route enabled in all generated helix configs (goldens refreshed), --require-feature-proof plumbed through run-and-verify.sh.
J transferred the fork from JasonVranek to Commit-Boost (GitHub leaves redirects, so existing clones keep working - but the canonical testing ground must not lean on redirects from a personal account). URL updated in .gitmodules and every doc reference; the submodule PIN is content-addressed and unchanged.
Adopt the tool-agnostic AGENTS.md convention for the agent orientation doc; keep a one-line CLAUDE.md stub so Claude Code still auto-loads it. Scrub session dates and a person+date submodule-transfer note from the orientation doc, and repoint its authoritative-docs router at the public design doc and the .agent/ back-room.
Move the internal backlog + plan proposals out of the public docs/ surface into .agent/ (agent context, not release docs): docs/SWEEP-BACKLOG.md -> .agent/SWEEP-BACKLOG.md and docs/plans/* -> .agent/plans/*. Scrub person-name attribution to generic maintainer language, drop session dates, and repoint the plans INDEX at the public docs and the new design doc.
…STAR.md docs/NORTH-STAR.md mixed a public thesis + 7 design laws (cited pervasively as "Law N") with an internal staged plan, ratified directions, and scars. Split it: - docs/DESIGN.md (public): the "what/why cb-testing is" thesis + the 7 numbered, quotable design laws, scrubbed of session attribution, with stable per-law anchors for citation. - .agent/NORTH-STAR.md (internal): the P0-P5 status ledger, architecture target, keep/kill list, and open scars. Delete the original docs/NORTH-STAR.md.
Update every reference to the moved docs so nothing dangles: - "NORTH-STAR Law N" / thesis citations -> docs/DESIGN.md (public). - docs/plans/* citations (ARCH, DEVELOPING, CHECKS, a genmodel doc-comment) -> .agent/plans/*. - SWEEP-BACKLOG citation in the helix config comment -> .agent/, date dropped. - Regenerate the golden config fixtures so the shared helix block matches the edited comment (the byte-identity oracle stays green). - Scrub a person+date submodule note and a session date from the public docs.
Keep the agent working notes on disk for local development but out of the public tree. Public docs that pointed into .agent/ now state the durable fact inline; AGENTS.md notes .agent/ is not present in a fresh clone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.