diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..8f81a9b70 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,166 @@ +# AGENTS.md + +Guidance for AI coding agents working in this repository. + +## What Commit-Boost is + +Commit-Boost is an Ethereum validator sidecar that standardizes how proposers make commitments +to third-party protocols, MEV-boost-style block building being the primary one. It is a Rust +workspace whose release artifact is the `commit-boost` binary, one subcommand per service, run +either as generated Docker services or as native processes. + +The services cooperate, wired together by one TOML config file: + +- **PBS service**: implements the Builder API toward the consensus client (header retrieval, + blinded-block submission, validator registration, status) and fans requests out to the + configured relays, applying selection, validation, and timing logic. Custom PBS builds can + replace the request/response logic while reusing the service scaffolding. +- **Signer service**: holds validator (consensus) keys and module-requested proxy keys, + generated behind the signer boundary. It exposes an authenticated HTTP API for signing + commitment data and never releases private keys; modules receive signatures and signed + delegation objects only. +- **Commit modules**: separate processes, first- or third-party, that implement commitment + protocols. They consume the SDK (the workspace's prelude library) to load config, talk to + the signer, and register metrics. Modules authenticate to the signer with per-module + pre-shared HMAC secrets from which short-lived JWTs are minted; separate admin credentials + protect the signer's administrative endpoints. + +`commit-boost init` reads the TOML config and generates a Docker Compose setup (services, +networks, an env file carrying generated secrets); each service also runs directly via its own +subcommand, with configuration passed through `CB_*` environment variables. Every service +exposes Prometheus metrics. + +## Repository map + +The authoritative workspace layout is `[workspace].members` in the root `Cargo.toml`. Roles: + +- `bin/`: the `commit-boost` binary and the prelude library that modules import. +- `crates/`: the library crates. Shared config, types, signing, and wire formats live in the + crate every other crate depends on (`common`); service logic lives in per-service crates; + compose generation lives in the CLI crate. +- `tests/`: the workspace integration suite (mock relays, validators, signer service). + Service-behavior regression tests go here; `just test` runs them. +- `benches/`: microbenchmarks and a Docker-based PBS load-benchmark harness (see the + justfile bench recipes). +- `examples/`: runnable module examples and config presets; docs snippets mirror these, so + changing an example implies checking the docs that quote it. +- `provisioning/`: Dockerfiles, the build container, the Helm chart, Grafana dashboards. +- `docs/`: the Docusaurus site. +- `api/`: the signer OpenAPI spec. The docs' API page renders it with SwaggerUI fetching the + main-branch copy from GitHub raw at page load, so spec changes are user-visible on merge and + the docs build does not validate the spec; unreleased behavior needs its marker inside the + spec's own description text. +- `.releases/`: one YAML per release naming the released commit; the anchor for every + version-line question (see Ground truth). +- `justfile`: the canonical developer commands; prefer its recipes over hand-rolled + invocations. + +## Working on the code + +- Format with `just fmt`, lint with `just clippy`, run the suite with `just test` (the + justfile pins its own toolchain and mirrors CI's lint invocation; the recipes are the source + of truth). Building requires `protoc`; `just install-protoc` provides it. `just build-bin + ` builds through the Docker build container, not the local toolchain; a plain local + build is `cargo build --release`. +- Match the existing code style: comments are minimal and state constraints, not narration; + error-binding names differ per crate, so follow the surrounding crate; struct field names + match wire names. +- Config keys are user-facing API. Only relay entries reject unknown keys (`[[relays]]`, + including relay entries inside `[[mux]]`); every other config section silently ignores + them, so a typo or an unreleased key outside a + relay entry does nothing rather than failing. Adding, renaming, or defaulting a key is a + compatibility decision, and `config.example.toml` plus the owning docs page change in the + same commit. +- Wire-visible strings (error bodies, log lines users are told to grep for, metric names and + labels) are documented surface: changing one means updating the docs pages and OpenAPI specs + that quote it. + +## Ground truth + +- The code is the source of truth. A documentation claim is wrong until the code proves it; + when reviewing or writing docs, ground every behavioral claim in a specific code location. +- Two lines always exist: the **released line** and **main**. The released commit is the + `commit:` field of the semver-newest file in `.releases/` (file names are exact tags, + including `-rcN` pre-releases; order by semantic version, never lexically or by mtime, since + `v0.9.*` sorts after `v0.10.*` as a string; naming rules in `.releases/README.md`). The docs + voice tracks the newest stable (non-rc) release. Release pins live on release-branch + lineages and are generally **not ancestors of main**, so never use `git merge-base + --is-ancestor` to decide whether a feature is released. Probe content instead: + `git show :`. +- While the docs site is unversioned (no `versioned_docs/` under `docs/`), one tree serves + users of the released binaries, so docs speak in the **released version's voice**. Behavior + that exists only on main is marked with a `:::info Unreleased` admonition (for sections) or + an inline marker naming the first version that will ship it, `(unreleased, from vX.Y)`, + stated once and tersely. When that release ships, deleting its markers is the whole docs + update. If the site gains versioned docs, per-version trees supersede this policy. +- `config.example.toml` is copied verbatim by users of released binaries. Unreleased keys + appear there **commented out**, with the unreleased marker in the comment: an uncommented + unreleased key either breaks released binaries at startup (inside relay entries) or is + silently ignored (everywhere else), and both mislead. +- Example and log values must be reproducible by a real binary: version strings exactly as + binaries self-report them, commit hashes equal to the release commit (annotated tag objects + are never printable by any build), timestamps/slots/block numbers arithmetically consistent + with each other, placeholder hosts from RFC 2606 (`example.com`), `https://` schemes, and + secrets shown truncated and obviously non-functional. + +## Writing style (docs and prose) + +- Plain operator prose, American English. No em dashes. No "Note that" lead-ins. No marketing + or filler vocabulary (leverage, seamlessly, robust, powerful, comprehensive, utilize). No + sentences that announce what the reader is about to read. +- Don't explain absences. When removing content, splice the neighbors and move on; never add + text justifying why something is no longer there. +- Terminology: "PBS service", "Signer service"; "commit module(s)" in running prose ("Commit + Modules" only in headings; never hyphenated). Heading case follows the page's dominant style. +- **One home per fact.** Every contract or nuance gets one full statement on the page that + owns its topic; every other mention is a one-line pointer to that home. Before adding a + fact, find where existing mentions of the topic already point and follow them. As of + writing: `configuration.md` owns config semantics (including hot reload, TLS, rate limiting, + keystore layouts); `mux-key-loaders.md` owns mux mechanics; `running/binary.md` owns + environment variables; `metrics-catalog.md` owns per-metric reference rows; + `troubleshooting.md` owns log walkthroughs and symptom tables; `developing/*` own SDK and + module-API contracts. `config.example.toml` comments stay one line per key plus a docs link: + it is the annotated reference for values, not a second prose home. +- Long runs of bold-label bullets are hard to scan; prefer prose or a real table. + +## Verifying changes + +- Docs build gate: run the docs build the way CI does (`.github/workflows/`; today + `npm install && npm run build` in `docs/`). Broken links fail the build; broken + anchors only warn, so also grep the build output case-insensitively for "broken anchor". +- Parse every fenced `toml`/`yaml`/`json` block in changed pages, and `config.example.toml` + itself (python `tomllib` / `yaml.safe_load`). +- Spellcheck prose with code spans and fences stripped. +- Examples must run. Prefer executing a documented flow (generate the compose setup from the + documented config, perform the documented auth round-trip) over reading it; executed + walkthroughs find breaks that reading does not. Check Rust snippets against the real SDK + signatures in the workspace or the `examples/` crates they mirror. + +## Trap classes + +Recurring bug shapes in this repository. Check for the class, not just the past instance. + +- **Released-vs-main drift**: a behavior claim may hold on only one line. Any claim about wire + formats, error bodies, config keys, or metrics needs checking against both the released + commit and main; cite the code location that supports it. +- **Secret vs token**: several environment variables hold HMAC *secrets* from which + short-lived tokens are minted. Calling them tokens produces curl examples that can never + authenticate. Verify what the receiving middleware actually validates before documenting an + auth flow. +- **Startup-frozen environment**: environment variables are read at process start; reload + endpoints cannot observe new values. Verify the actual data flow before documenting any + hot-rotation or reload pattern, and state what a reload can and cannot pick up. +- **Silently ignored config**: outside relay entries, unknown config keys are accepted and + dropped, so a misplaced or misspelled key looks configured while doing nothing. Verify a key + is consumed at the nesting level where the docs place it. +- **Dead knobs**: config fields, chart values, or flags consumed by no code path (Helm + templates are especially prone). Verify each documented option is actually read before + listing it. +- **Documented intent, not behavior**: comments and older docs sometimes describe features + that were planned but never implemented. Trust only the code path you can cite. + +## Maintenance of this file + +This file holds only orientation, conventions, methods, and trap classes, so it stays valid as +features are added. If an edit records a fact about a specific version or feature, that fact +belongs in the docs themselves, a code comment, or the issue tracker instead. diff --git a/api/signer-api.yml b/api/signer-api.yml index be44f8fdd..30b1792a0 100644 --- a/api/signer-api.yml +++ b/api/signer-api.yml @@ -2,7 +2,37 @@ openapi: "3.1.1" info: title: Signer API version: "0.2.0" - description: API that allows commit modules to request generic signatures from validators + description: | + API that allows commit modules to request generic signatures from validators. + + ## Authentication + + All endpoints (except `/status`) require a Bearer JWT in the `Authorization` header. + A missing or malformed `Authorization` header returns `400`, not `401`. + + ### Module JWT claims + + HS256-signed token with the module's pre-shared secret (`CB_SIGNER_JWT` env var): + + - **module** (string, required): Module `id` from `[[modules]]` in `cb-config.toml`. + - **route** (string, required): Exact request path, e.g. `/signer/v1/get_pubkeys`. + - **exp** (integer, required): UNIX expiry timestamp. The SDK issues tokens with a 5-minute expiry; the server only validates that the token is unexpired. + - **payload_hash** (string, POST only): Keccak-256 hash of the JSON request body, `0x`-prefixed. Prevents JWT replay attacks. + + Refresh is client-side: the module generates a new JWT locally. No refresh endpoint. + + ### Admin JWT + + Admin endpoints (`/reload`, `/revoke_jwt`) use a separate HS256 secret (`CB_SIGNER_ADMIN_JWT` env var) with the following claims: + + - **admin** (boolean, required): Must be `true`. + - **route** (string, required): Exact request path, `/reload` or `/revoke_jwt`. + - **exp** (integer, required): UNIX expiry timestamp. + - **payload_hash** (string, required): Keccak-256 hash of the JSON request body, `0x`-prefixed. Both admin endpoints require a JSON body, so this claim is required in practice. + + ### Rate limiting + + Per-IP rate limit on JWT auth failures. Default: 3 failures within 5 minutes. Configurable via `[signer].jwt_auth_fail_limit` and `jwt_auth_fail_timeout_seconds` in `cb-config.toml`. tags: - name: Signer - name: Management @@ -10,13 +40,8 @@ paths: /signer/v1/get_pubkeys: get: summary: Get a list of public keys for which signatures may be requested - description: > - This endpoint requires a valid JWT Bearer token. - - The token **must include** the following claims: - - `exp` (integer): Expiration timestamp - - `route` (string): The route being requested (must be `/signer/v1/get_pubkeys` for this endpoint). - - `module` (string): The ID of the module making the request, which must match a module ID in the Commit-Boost configuration file. + description: | + Requires a module JWT (see Authentication above); the `route` claim must be `/signer/v1/get_pubkeys`. tags: - Signer security: @@ -48,34 +73,25 @@ paths: type: array items: $ref: "#/components/schemas/EcdsaAddress" - "500": - description: Internal error + "400": + description: The request did not include a valid `Authorization` header with a Bearer token, or the client IP could not be determined. content: - application/json: + text/plain: schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 500 - message: - type: string - example: "Internal error" + type: string + example: "Header of type `authorization` was missing" + "401": + $ref: "#/components/responses/UnauthorizedJwt" + "429": + $ref: "#/components/responses/RateLimited" + "500": + $ref: "#/components/responses/InternalError" /signer/v1/request_signature/bls: post: summary: Request a signature for a 32-byte blob of data (typically a hash), signed by the BLS private key for the requested public key. - description: > - This endpoint requires a valid JWT Bearer token. - - The token **must include** the following claims: - - `exp` (integer): Expiration timestamp - - `module` (string): The ID of the module making the request, which must match a module ID in the Commit-Boost configuration file. - - `route` (string): The route being requested (must be `/signer/v1/request_signature/bls` for this endpoint). - - `payload_hash` (string): The Keccak-256 hash of the JSON-encoded request body, with optional `0x` prefix. This is required to prevent JWT replay attacks. + description: | + Requires a module JWT (see Authentication above); the `route` claim must be `/signer/v1/request_signature/bls`. tags: - Signer security: @@ -89,16 +105,17 @@ paths: required: [pubkey, object_root, nonce] properties: pubkey: - description: The 48-byte BLS public key, with optional `0x` prefix, of the proposer key that you want to request a signature from. + description: The consensus (proposer) pubkey to sign with. $ref: "#/components/schemas/BlsPubkey" object_root: - description: The 32-byte data you want to sign, with optional `0x` prefix. + description: The 32-byte digest to sign. $ref: "#/components/schemas/B256" nonce: $ref: "#/components/schemas/Nonce" example: pubkey: "0xa3ffa9241f78279f1af04644cb8c79c2d8f02bcf0e28e2f186f6dcccac0a869c2be441fda50f0dea895cfce2e53f0989" object_root: "0x3e9f4a78b5c21d64f0b8e3d9a7f5c02b4d1e67a3c8f29b5d6e4a3b1c8f72e6d9" + nonce: 1 responses: "200": description: A successful signature response. @@ -110,120 +127,44 @@ paths: pubkey: "0x883827193f7627cd04e621e1e8d56498362a52b2a30c9a1c72036eb935c4278dee23d38a24d2f7dda62689886f0c39f4" object_root: "0x0123456789012345678901234567890123456789012345678901234567890123" module_signing_id: "0x6a33a23ef26a4836979edff86c493a69b26ccf0b4a16491a815a13787657431b" + nonce: 1 + chain_id: "0x1" signature: "0xa43e623f009e615faa3987368f64d6286a4103de70e9a81d82562c50c91eae2d5d6fb9db9fe943aa8ee42fd92d8210c1149f25ed6aa72a557d74a0ed5646fdd0e8255ec58e3e2931695fe913863ba0cdf90d29f651bce0a34169a6f6ce5b3115" "400": description: | This can occur in several scenarios: + - The request did not include a valid `Authorization` header with a Bearer token. - The Commit-Boost configuration file does not contain a signing ID for the module that made the request. - - You requested an operation while using the Dirk signer mode instead of locally-managed signer mode, but Dirk doesn't support that operation. - Something went wrong while preparing your request; the error text will provide more information. content: - application/json: + text/plain: schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 400 - message: - type: string - example: "Bad request: Invalid pubkey format" + type: string + example: "bad request: Module signing ID not found" "401": - description: The requesting module did not provide a JWT string in the request's authorization header, or the JWT string was not configured in the signer service's configuration file as belonging to the module. - content: - application/json: - schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 401 - message: - type: string - example: "Unauthorized" - + $ref: "#/components/responses/UnauthorizedJwt" + "404": description: You either requested a route that doesn't exist, or you requested a signature from a key that does not exist. content: - application/json: + text/plain: schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 404 - message: - type: string - example: "Unknown pubkey" + type: string + example: "unknown consensus signer: 0xa9e9cff900de07e295a044789fd4bdb6785eb0651ad282f9e76d12afd87e75180bdd64caf2e315b815d7322bd31ab48a" + "422": + $ref: "#/components/responses/DeserializationFailed" "429": - description: Your module attempted and failed JWT authentication too many times recently, and is currently timed out. It cannot make any more requests until the timeout ends. - content: - application/json: - schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 429 - message: - type: string - example: "Too many requests" + $ref: "#/components/responses/RateLimited" "500": - description: Your request was valid, but something went wrong internally that prevented it from being fulfilled. - content: - application/json: - schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 500 - message: - type: string - example: "Internal error" + $ref: "#/components/responses/InternalError" "502": - description: The signer service is running in Dirk signer mode, but Dirk could not be reached. - content: - application/json: - schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 502 - message: - type: string - example: "Bad gateway: Dirk signer service is unreachable" + $ref: "#/components/responses/DirkUnreachable" /signer/v1/request_signature/proxy-bls: post: summary: Request a signature for a 32-byte blob of data (typically a hash), signed by the BLS private key for the requested proxy public key. - description: > - This endpoint requires a valid JWT Bearer token. - - The token **must include** the following claims: - - `exp` (integer): Expiration timestamp - - `module` (string): The ID of the module making the request, which must match a module ID in the Commit-Boost configuration file. - - `route` (string): The route being requested (must be `/signer/v1/request_signature/proxy-bls` for this endpoint). - - `payload_hash` (string): The Keccak-256 hash of the JSON-encoded request body, with optional `0x` prefix. This is required to prevent JWT replay attacks. + description: | + Requires a module JWT (see Authentication above); the `route` claim must be `/signer/v1/request_signature/proxy-bls`. tags: - Signer security: @@ -237,16 +178,17 @@ paths: required: [proxy, object_root, nonce] properties: proxy: - description: The 48-byte BLS public key (for `proxy_bls` mode) or the 20-byte Ethereum address (for `proxy_ecdsa` mode), with optional `0x` prefix, of the proxy key that you want to request a signature from. + description: The proxy pubkey to sign with. $ref: "#/components/schemas/BlsPubkey" object_root: - description: The 32-byte data you want to sign, with optional `0x` prefix. + description: The 32-byte digest to sign. $ref: "#/components/schemas/B256" nonce: $ref: "#/components/schemas/Nonce" example: - pubkey: "0xa3ffa9241f78279f1af04644cb8c79c2d8f02bcf0e28e2f186f6dcccac0a869c2be441fda50f0dea895cfce2e53f0989" + proxy: "0xa3ffa9241f78279f1af04644cb8c79c2d8f02bcf0e28e2f186f6dcccac0a869c2be441fda50f0dea895cfce2e53f0989" object_root: "0x3e9f4a78b5c21d64f0b8e3d9a7f5c02b4d1e67a3c8f29b5d6e4a3b1c8f72e6d9" + nonce: 1 responses: "200": description: A successful signature response. @@ -258,120 +200,44 @@ paths: pubkey: "0x883827193f7627cd04e621e1e8d56498362a52b2a30c9a1c72036eb935c4278dee23d38a24d2f7dda62689886f0c39f4" object_root: "0x0123456789012345678901234567890123456789012345678901234567890123" module_signing_id: "0x6a33a23ef26a4836979edff86c493a69b26ccf0b4a16491a815a13787657431b" + nonce: 1 + chain_id: "0x1" signature: "0xa43e623f009e615faa3987368f64d6286a4103de70e9a81d82562c50c91eae2d5d6fb9db9fe943aa8ee42fd92d8210c1149f25ed6aa72a557d74a0ed5646fdd0e8255ec58e3e2931695fe913863ba0cdf90d29f651bce0a34169a6f6ce5b3115" "400": description: | This can occur in several scenarios: + - The request did not include a valid `Authorization` header with a Bearer token. - The Commit-Boost configuration file does not contain a signing ID for the module that made the request. - - You requested an operation while using the Dirk signer mode instead of locally-managed signer mode, but Dirk doesn't support that operation. - Something went wrong while preparing your request; the error text will provide more information. content: - application/json: + text/plain: schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 400 - message: - type: string - example: "Bad request: Invalid pubkey format" + type: string + example: "bad request: Module signing ID not found" "401": - description: The requesting module did not provide a JWT string in the request's authorization header, or the JWT string was not configured in the signer service's configuration file as belonging to the module. - content: - application/json: - schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 401 - message: - type: string - example: "Unauthorized" - + $ref: "#/components/responses/UnauthorizedJwt" + "404": description: You either requested a route that doesn't exist, or you requested a signature from a key that does not exist. content: - application/json: + text/plain: schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 404 - message: - type: string - example: "Unknown pubkey" + type: string + example: "unknown proxy signer: 0xa9e9cff900de07e295a044789fd4bdb6785eb0651ad282f9e76d12afd87e75180bdd64caf2e315b815d7322bd31ab48a" + "422": + $ref: "#/components/responses/DeserializationFailed" "429": - description: Your module attempted and failed JWT authentication too many times recently, and is currently timed out. It cannot make any more requests until the timeout ends. - content: - application/json: - schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 429 - message: - type: string - example: "Too many requests" + $ref: "#/components/responses/RateLimited" "500": - description: Your request was valid, but something went wrong internally that prevented it from being fulfilled. - content: - application/json: - schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 500 - message: - type: string - example: "Internal error" + $ref: "#/components/responses/InternalError" "502": - description: The signer service is running in Dirk signer mode, but Dirk could not be reached. - content: - application/json: - schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 502 - message: - type: string - example: "Bad gateway: Dirk signer service is unreachable" + $ref: "#/components/responses/DirkUnreachable" /signer/v1/request_signature/proxy-ecdsa: post: summary: Request a signature for a 32-byte blob of data (typically a hash), signed by the ECDSA private key for the requested proxy Ethereum address. - description: > - This endpoint requires a valid JWT Bearer token. - - The token **must include** the following claims: - - `exp` (integer): Expiration timestamp - - `module` (string): The ID of the module making the request, which must match a module ID in the Commit-Boost configuration file. - - `route` (string): The route being requested (must be `/signer/v1/request_signature/proxy-ecdsa` for this endpoint). - - `payload_hash` (string): The Keccak-256 hash of the JSON-encoded request body, with optional `0x` prefix. This is required to prevent JWT replay attacks. + description: | + Requires a module JWT (see Authentication above); the `route` claim must be `/signer/v1/request_signature/proxy-ecdsa`. tags: - Signer security: @@ -385,16 +251,17 @@ paths: required: [proxy, object_root, nonce] properties: proxy: - description: The 20-byte Ethereum address, with optional `0x` prefix, of the proxy key that you want to request a signature from. + description: The proxy address to sign with. $ref: "#/components/schemas/EcdsaAddress" object_root: - description: The 32-byte data you want to sign, with optional `0x` prefix. + description: The 32-byte digest to sign. $ref: "#/components/schemas/B256" nonce: $ref: "#/components/schemas/Nonce" example: proxy: "0x71f65e9f6336770e22d148bd5e89b391a1c3b0bb" object_root: "0x3e9f4a78b5c21d64f0b8e3d9a7f5c02b4d1e67a3c8f29b5d6e4a3b1c8f72e6d9" + nonce: 1 responses: "200": description: A successful signature response. @@ -406,120 +273,43 @@ paths: address: "0x71f65e9f6336770e22d148bd5e89b391a1c3b0bb" object_root: "0x3e9f4a78b5c21d64f0b8e3d9a7f5c02b4d1e67a3c8f29b5d6e4a3b1c8f72e6d9" module_signing_id: "0x6a33a23ef26a4836979edff86c493a69b26ccf0b4a16491a815a13787657431b" + nonce: 1 + chain_id: "0x1" signature: "0x985b495f49d1b96db3bba3f6c5dd1810950317c10d4c2042bd316f338cdbe74359072e209b85e56ac492092d7860063dd096ca31b4e164ef27e3f8d508e656801c" "400": description: | This can occur in several scenarios: + - The request did not include a valid `Authorization` header with a Bearer token. - The Commit-Boost configuration file does not contain a signing ID for the module that made the request. - You requested an operation while using the Dirk signer mode instead of locally-managed signer mode, but Dirk doesn't support that operation. - Something went wrong while preparing your request; the error text will provide more information. content: - application/json: + text/plain: schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 400 - message: - type: string - example: "Bad request: Invalid pubkey format" + type: string + example: "Dirk signer does not support this operation" "401": - description: The requesting module did not provide a JWT string in the request's authorization header, or the JWT string was not configured in the signer service's configuration file as belonging to the module. - content: - application/json: - schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 401 - message: - type: string - example: "Unauthorized" - + $ref: "#/components/responses/UnauthorizedJwt" + "404": description: You either requested a route that doesn't exist, or you requested a signature from a key that does not exist. content: - application/json: + text/plain: schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 404 - message: - type: string - example: "Unknown pubkey" + type: string + example: "unknown proxy signer: 0x71f65e9f6336770e22d148bd5e89b391a1c3b0bb" + "422": + $ref: "#/components/responses/DeserializationFailed" "429": - description: Your module attempted and failed JWT authentication too many times recently, and is currently timed out. It cannot make any more requests until the timeout ends. - content: - application/json: - schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 429 - message: - type: string - example: "Too many requests" + $ref: "#/components/responses/RateLimited" "500": - description: Your request was valid, but something went wrong internally that prevented it from being fulfilled. - content: - application/json: - schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 500 - message: - type: string - example: "Internal error" - "502": - description: The signer service is running in Dirk signer mode, but Dirk could not be reached. - content: - application/json: - schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 502 - message: - type: string - example: "Bad gateway: Dirk signer service is unreachable" + $ref: "#/components/responses/InternalError" /signer/v1/generate_proxy_key: post: summary: Request a proxy key be generated for a specific consensus pubkey - description: > - This endpoint requires a valid JWT Bearer token. - - The token **must include** the following claims: - - `exp` (integer): Expiration timestamp - - `module` (string): The ID of the module making the request, which must match a module ID in the Commit-Boost configuration file. - - `route` (string): The route being requested (must be `/signer/v1/generate_proxy_key` for this endpoint). - - `payload_hash` (string): The Keccak-256 hash of the JSON-encoded request body, with optional `0x` prefix. This is required to prevent JWT replay attacks. + description: | + Requires a module JWT (see Authentication above); the `route` claim must be `/signer/v1/generate_proxy_key`. tags: - Signer security: @@ -586,38 +376,164 @@ paths: delegator: "0xa9e9cff900de07e295a044789fd4bdb6785eb0651ad282f9e76d12afd87e75180bdd64caf2e315b815d7322bd31ab48a" proxy: "0x71f65e9f6336770e22d148bd5e89b391a1c3b0bb" signature: "0xb5b5b71d1701cc45086af3d3d86bf9d3c509442835e5b9f7734923edc9a6c538e743d70613cdef90b7e5b171fbbe6a29075b3f155e4bd66d81ff9dbc3b6d7fa677d169b2ceab727ffa079a31fe1fc0e478752e9da9566a9408e4db24ac6104db" + "400": + description: | + This can occur in several scenarios: + - The request did not include a valid `Authorization` header with a Bearer token. + - You requested an ECDSA proxy key while using the Dirk signer mode, but Dirk only supports BLS operations. + - Something went wrong while preparing your request; the error text will provide more information. + content: + text/plain: + schema: + type: string + example: "Dirk signer does not support this operation" + "401": + $ref: "#/components/responses/UnauthorizedJwt" "404": description: Unknown value (pubkey, etc.) content: - application/json: + text/plain: schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 404 - message: - type: string - example: "Unknown pubkey" + type: string + example: "unknown consensus signer: 0xa9e9cff900de07e295a044789fd4bdb6785eb0651ad282f9e76d12afd87e75180bdd64caf2e315b815d7322bd31ab48a" + "422": + $ref: "#/components/responses/DeserializationFailed" + "429": + $ref: "#/components/responses/RateLimited" + "500": + $ref: "#/components/responses/InternalError" + "502": + $ref: "#/components/responses/DirkUnreachable" + + /reload: + post: + summary: Hot-reload signer configuration + description: | + Re-reads cb-config.toml and environment variables, rebuilding the signer's + internal state. Accepts optional body overrides for JWT secrets and the + admin secret. + + **Behavior:** + - New modules in config are registered. + - Removed modules are dropped from the access list. + - JWT secrets and admin secret are reset to env var values. + - Previous runtime changes (from /revoke_jwt or body overrides) are reverted. + + **Body overrides** (applied on top of the config baseline): + - `jwt_secrets`: comma-separated `=` pairs. + - `admin_secret`: string to override the admin JWT secret. + + Body overrides are **not persisted** across restarts. + + A JSON body with `Content-Type: application/json` is required even when no + overrides are sent; send an empty object `{}`. + tags: + - Management + security: + - AdminBearerAuth: [] + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + jwt_secrets: + description: Comma-separated list of MODULE_ID=SECRET pairs to override module JWT secrets + type: string + example: "module_a=newsecret,module_b=anothersecret" + admin_secret: + description: Override for the admin JWT secret + type: string + example: "my-new-admin-secret" + responses: + "200": + description: Configuration reloaded successfully + "400": + description: The body references a module ID not present in the config, or the request did not include a valid `Authorization` header with a Bearer token. + content: + text/plain: + schema: + type: string + example: "bad request: Module unknown-module not found in config, cannot override JWT secret" + "401": + $ref: "#/components/responses/UnauthorizedJwt" + "422": + $ref: "#/components/responses/DeserializationFailed" + "429": + $ref: "#/components/responses/RateLimited" "500": - description: Internal error + description: Failed to reload config (previous state preserved) content: - application/json: + text/plain: schema: - type: object - required: - - code - - message - properties: - code: - type: number - example: 500 - message: - type: string - example: "Internal error" + type: string + example: "internal error" + + /revoke_jwt: + post: + summary: Immediately revoke a module's access + description: | + Removes a module from the signer's access list. The module will no longer + be able to authenticate with its JWT. + + If the module is still present in cb-config.toml, the next `/reload` will + re-add it. Remove the module from the config to make revocation permanent. + tags: + - Management + security: + - AdminBearerAuth: [] + requestBody: + required: true + content: + application/json: + schema: + type: object + required: + - module_id + properties: + module_id: + description: The ID of the module to revoke + type: string + example: "MY_MODULE" + responses: + "200": + description: Module access revoked successfully + "400": + description: The request did not include a valid `Authorization` header with a Bearer token. + content: + text/plain: + schema: + type: string + example: "Header of type `authorization` was missing" + "401": + $ref: "#/components/responses/UnauthorizedJwt" + "404": + description: Module ID not found + content: + text/plain: + schema: + type: string + example: "module id not found" + "422": + $ref: "#/components/responses/DeserializationFailed" + "429": + $ref: "#/components/responses/RateLimited" + + /status: + get: + summary: Health check + description: Simple health check endpoint. Returns 200 OK with no body. No authentication required. + tags: + - Management + responses: + "200": + description: Signer service is healthy + content: + text/plain: + schema: + type: string + example: "" components: securitySchemes: @@ -625,6 +541,46 @@ components: type: http scheme: bearer bearerFormat: JWT + AdminBearerAuth: + type: http + scheme: bearer + bearerFormat: JWT + responses: + UnauthorizedJwt: + description: The JWT provided in the request's authorization header was invalid. For example, it was expired, signed with the wrong secret, or its claims did not match the request. + content: + text/plain: + schema: + type: string + example: "unauthorized" + RateLimited: + description: Your client IP failed JWT authentication too many times recently, and is currently timed out. It cannot make any more requests until the timeout ends. + content: + text/plain: + schema: + type: string + example: "rate limited for 12.3" + DeserializationFailed: + description: The request body could not be deserialized. For example, a field was not valid hex of the expected length. + content: + text/plain: + schema: + type: string + example: "Failed to deserialize the JSON body into the target type" + InternalError: + description: Internal server error. The request was valid but could not be fulfilled. + content: + text/plain: + schema: + type: string + example: "internal error" + DirkUnreachable: + description: The signer service is running in Dirk signer mode, but Dirk could not be reached. + content: + text/plain: + schema: + type: string + example: "Dirk communication error" schemas: B256: type: string @@ -666,11 +622,12 @@ components: nonce: $ref: "#/components/schemas/Nonce" chain_id: - description: The chain ID that the signature is valid for, as specified in the Commit-Boost configuration - type: integer - example: 1 + description: The chain ID that the signature is valid for, as specified in the Commit-Boost configuration. Serialized as a `0x`-prefixed hex quantity string. + type: string + format: hex + example: "0x1" signature: - description: The BLS signature of the Merkle root hash of the provided `object_root` field and the requesting module's Signing ID. For details on this signature, see the [signature structure documentation](https://commit-boost.github.io/commit-boost-client/developing/prop-commit-signing.md#structure-of-a-signature). + description: The BLS signature of the SSZ root over the provided `object_root`, the module's signing ID, the nonce, and the chain ID (`PropCommitSigningInfo`), domain-separated per the [signature structure documentation](https://commit-boost.github.io/commit-boost-client/developing/prop-commit-signing#structure-of-a-signature). $ref: "#/components/schemas/BlsSignature" EcdsaSignatureResponse: type: object @@ -687,20 +644,23 @@ components: nonce: $ref: "#/components/schemas/Nonce" chain_id: - description: The chain ID that the signature is valid for, as specified in the Commit-Boost configuration - type: integer - example: 1 + description: The chain ID that the signature is valid for, as specified in the Commit-Boost configuration. Serialized as a `0x`-prefixed hex quantity string. + type: string + format: hex + example: "0x1" signature: - description: The ECDSA signature (in Ethereum RSV format) of the Merkle root hash of the provided `object_root` field and the requesting module's Signing ID. For details on this signature, see the [signature structure documentation](https://commit-boost.github.io/commit-boost-client/developing/prop-commit-signing.md#structure-of-a-signature). + description: The ECDSA signature (in Ethereum RSV format) of the SSZ root over the provided `object_root`, the module's signing ID, the nonce, and the chain ID (`PropCommitSigningInfo`), domain-separated per the [signature structure documentation](https://commit-boost.github.io/commit-boost-client/developing/prop-commit-signing#structure-of-a-signature). $ref: "#/components/schemas/EcdsaSignature" Nonce: type: integer description: | Replay-protection nonce, always mixed into the signing root via `PropCommitSigningInfo`. It - must be an unsigned 64-bit integer between 0 and 2^64-2 (18446744073709551614), inclusive. + is an unsigned 64-bit integer. Per the [EIP-2681](https://eips.ethereum.org/EIPS/eip-2681) + convention the maximum is 2^64-2 (18446744073709551614), but the service does not enforce + this cap. Modules that track nonces for replay protection should use a monotonically increasing value per key. Modules that do not use replay protection should always send `0`. minimum: 0 - maximum: 18446744073709551614 + maximum: 18446744073709551615 example: 1 diff --git a/config.example.toml b/config.example.toml index 4b6b2853f..838312480 100644 --- a/config.example.toml +++ b/config.example.toml @@ -4,15 +4,15 @@ # Chain spec ID. Supported values: # A network ID. Supported values: Mainnet, Holesky, Sepolia, Hoodi. Lower case values e.g. "mainnet" are also accepted # A custom object, e.g., chain = { genesis_time_secs = 1695902400, path = "/path/to/spec.json" }, with a path to a chain spec file, either in .json format (e.g., as returned by the beacon endpoint /eth/v1/config/spec), or in .yml format (see examples in tests/data). -# A custom object, e.g., chain = { genesis_time_secs = 1695902400, slot_time_secs = 12, genesis_fork_version = "0x01017000", fulu_fork_slot = 5283840, chain_id = 17000 }. +# A custom object, e.g., chain = { genesis_time_secs = 1695902400, slot_time_secs = 12, genesis_fork_version = "0x01017000", fulu_fork_slot = 5283840, chain_id = 17000 }. All fields are required. chain = "Holesky" -# Configuration for the PBS module +# Configuration for the PBS service [pbs] -# Docker image to use for the PBS module. +# Docker image to use for the PBS service. # OPTIONAL, DEFAULT: ghcr.io/commit-boost/commit-boost:latest docker_image = "ghcr.io/commit-boost/commit-boost:latest" -# Whether to enable the PBS module to request signatures from the Signer module (not used in the default PBS image) +# Whether to enable the PBS service to request signatures from the Signer service (not used in the default PBS image) # OPTIONAL, DEFAULT: false with_signer = false # Host to receive BuilderAPI calls from beacon node @@ -62,7 +62,7 @@ extra_validation_enabled = false # a fallback if the user's own SSV node is not reachable. # OPTIONAL, DEFAULT: "https://api.ssv.network/api/v4/" # ssv_public_api_url = "https://api.ssv.network/api/v4/" -# Timeout for any HTTP requests sent from the PBS module to other services, in seconds +# Timeout for any HTTP requests sent from the PBS service to other services, in seconds # OPTIONAL, DEFAULT: 10 http_timeout_seconds = 10 # Maximum number of retries for validator registrations per relay @@ -79,7 +79,7 @@ validator_registration_batch_size = "" # OPTIONAL, DEFAULT: 384 mux_registry_refresh_interval_seconds = 384 -# The PBS module needs one or more [[relays]] as defined below. +# The PBS service needs one or more [[relays]] as defined below. [[relays]] # Relay ID to use in telemetry # OPTIONAL, DEFAULT: URL hostname @@ -180,14 +180,13 @@ timeout_get_header_ms = 900 id = "mux-relay-1" url = "http://0xa119589bb33ef52acbb8116832bec2b58fca590fe5c85eac5d3230b44d5bc09fe73ccd21f88eab31d6de16194d17782e@def.xyz" -# Configuration for the Signer Module, only required if any `commit` module is present, or if `pbs.with_signer = true` -# Currently three types of Signer modules are supported (only one can be used at a time): -# - Remote: a remote Web3Signer instance +# Configuration for the Signer service, only required if any `commit` module is present, or if `pbs.with_signer = true` +# Currently two types of Signer service are supported (only one can be used at a time): # - Dirk: a remote Dirk instance -# - Local: a local Signer module -# More details on the docs (https://commit-boost.github.io/commit-boost-client/get_started/configuration/#signer-module) +# - Local: a local Signer service +# More details on the docs (https://commit-boost.github.io/commit-boost-client/get_started/configuration/#signer-service) [signer] -# Docker image to use for the Signer module. +# Docker image to use for the Signer service. # OPTIONAL, DEFAULT: ghcr.io/commit-boost/commit-boost:latest docker_image = "ghcr.io/commit-boost/commit-boost:latest" # Host to bind the Signer API server to @@ -221,12 +220,14 @@ jwt_auth_fail_timeout_seconds = 300 # [signer.tls_mode] # How to use TLS for the Signer's HTTP server; two modes are supported: # - type = "insecure": disable TLS, so the server runs in HTTP mode (not recommended for production). -# - type = "certificate": Use TLS. Include a property named "path" below this with the provided path; `path` should be a directory containing `cert.pem` and `key.pem` files to use. If they don't exist, they'll be automatically generated in self-signed mode. +# - type = "certificate": Use TLS. Include a property named "path" below this with the provided path; `path` should be a directory containing existing `cert.pem` and `key.pem` files to use. They are NOT auto-generated: the Signer fails to start if either file is missing. # OPTIONAL, DEFAULT: -# type = "certificate" -# path = "./certs" +# type = "insecure" # For Remote signer: +# NOT IMPLEMENTED. The `[signer.remote]` (Web3Signer) variant parses, but the Signer service +# rejects it at startup with `Remote signer configured`. Dirk is currently the only supported +# remote signer; use `[signer.dirk]` below. # [signer.remote] # URL of the Web3Signer instance # url = "https://remote.signer.url" @@ -253,20 +254,20 @@ jwt_auth_fail_timeout_seconds = 300 # Complete URL of a Dirk gateway # url = "https://localhost:8881" # Wallets to load consensus keys from -# accounts = ["Wallet1", "DistributedWallet"] +# wallets = ["Wallet1", "DistributedWallet"] # [[signer.dirk.hosts]] # server_name = "localhost-2" # url = "https://localhost:8882" -# accounts = ["Wallet2", "DistributedWallet"] +# wallets = ["Wallet2", "DistributedWallet"] -# Configuration for how the Signer module should store proxy delegations. +# Configuration for how the Signer service should store proxy delegations. # OPTIONAL # [signer.dirk.store] # proxy_dir = "/path/to/proxies" # For Local signer: -# Configuration for how the Signer module should load validator keys. Currently two types of loaders are supported: +# Configuration for how the Signer service should load validator keys. Currently two types of loaders are supported: # - File: load keys from a plain text file (unsafe, use only for testing purposes) # - ValidatorsDir: load keys from a `keys` and `secrets` file/folder (ERC-2335 style keystores). More details can be found in the docs (https://commit-boost.github.io/commit-boost-client/get_started/configuration/) [signer.local.loader] @@ -288,7 +289,7 @@ key_path = "./tests/data/keys.example.json" # For lodestar, it's the path to the file containing the decryption password. # For nimbus, it's the path to the directory where the `` files are located. # secrets_path = "" -# Configuration for how the Signer module should store proxy delegations. Supported types of store are: +# Configuration for how the Signer service should store proxy delegations. Supported types of store are: # - File: store keys and delegations from a plain text file (unsafe, use only for testing purposes) # - ERC2335: store keys and delegations safely using ERC-2335 style keystores. More details can be found in the docs (https://commit-boost.github.io/commit-boost-client/get_started/configuration#proxy-keys-store) # OPTIONAL, if missing proxies are lost on restart diff --git a/docs/docs/architecture/overview.md b/docs/docs/architecture/overview.md index 20137675a..c00e09a2b 100644 --- a/docs/docs/architecture/overview.md +++ b/docs/docs/architecture/overview.md @@ -4,12 +4,12 @@ description: Overview of the architecture of Commit-Boost # Overview -Below is schematic overview of Commit-Boost. +Below is a schematic overview of Commit-Boost. Commit-Boost runs as a single sidecar composed of multiple modules: -- Pbs Module with the [BuilderAPI](https://ethereum.github.io/builder-specs/) for [MEV Boost](https://docs.flashbots.net/flashbots-mev-boost/architecture-overview/specifications) -- A Signer Module implementing the SignerAPI -- Commit Modules that implement some custom commit protocol logic +- A PBS service with the [BuilderAPI](https://ethereum.github.io/builder-specs/) for [MEV Boost](https://docs.flashbots.net/flashbots-mev-boost/architecture-overview/specifications) +- A Signer service implementing the SignerAPI +- Commit modules that implement some custom commit protocol logic - Telemetry modules like Prometheus and Grafana ![architecture](./img/architecture.png) diff --git a/docs/docs/developing/commit-module.md b/docs/docs/developing/commit-module.md deleted file mode 100644 index 30921be2f..000000000 --- a/docs/docs/developing/commit-module.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -sidebar_position: 2 ---- - -# Commit Module - -While a module can be written in any language, we currently provide some utilities for Rust, with the goal of supporting more generalized APIs and simplify development in languages other than Rust. - -In Rust, we provide utilities to load and run modules. Simply add to your `Cargo.toml`: -```toml -commit-boost = { git = "https://github.com/Commit-Boost/commit-boost-client", version = "..." } -``` - -You will now be able to import the utils with: -```rust -use commit_boost::prelude::*; -``` - - -## Config -Your module will likely need a configuration for the Node Operator to customize. This will have to be in the `cb-config.toml` file, in the correct `[[module]]` section. In the module, you can define and load your config as follows. - -First, define all the parameters needed in a struct: -```rust -#[derive(Debug, Deserialize)] -struct ExtraConfig { - sleep_secs: u64, -} -``` -then pass that struct to the `load_commit_module_config` function, which will load and parse the config. Your custom config will be under the `extra` field. - -```rust -let config = load_commit_module_config::().unwrap(); -let to_sleep = config.extra.sleep_secs; -``` - -The loaded `config` also has a few other useful fields: -- the unique `id` of the module -- chain spec -- a `SignerClient` to call the [SignerAPI](/api), already setup with the correct JWT - - -## Requesting signatures -At its core the Signer Module simply provides a signature on a 32-byte data digest. The signatures are currently provided with either the validator keys (BLS) or a proxy key (BLS or ECDSA) for a given validator key, both on the [builder domain](https://github.com/Commit-Boost/commit-boost-client/blob/main/crates/common/src/signature.rs#L88-L96). - -In the example we use `TreeHash`, already used in the CL, to create the digest from a custom struct: -```rust -#[derive(TreeHash)] -struct Datagram { - data: u64, -} -``` - -Furthermore, in order to request a signature, we'd need a public key of the validator. You can get a list of available keys by calling: -```rust -let pubkeys = config.signer_client.get_pubkeys().await.unwrap(); -``` - -Which will call the `get_pubkeys` endpoint of the [SignerAPI](/api), returning all the consensus pubkeys and the corresponding proxy keys, of your module. - -Note that the requests are authenticated using a JWT, that must be regularly refreshed as it expires after a certain time. To do so, you can call: -```rust -config.signer_client.refresh_token().await.unwrap(); -``` -You have the `SIGNER_JWT_EXPIRATION` constant available in the `commit-boost` crate, which is the time in seconds after which the JWT will expire. - -Then, we can request a signature either with a consensus key or with a proxy key: - -### With a consensus key -Requesting a signature is as simple as: -```rust -let datagram = Datagram { data: 1 }; -let request = SignConsensusRequest::builder(pubkey).with_msg(&datagram); -let signature = config.signer_client.request_consensus_signature(&request).await.unwrap(); -``` - -Where `pubkey` is the validator (consensus) public key for which the signature is requested. - -### With a proxy key -You'll have to first request a proxy key be generated for a given consensus key. -We support two signature schemes for proxies: BLS or ECDSA. - -To request a proxy: -```rust -// BLS proxy -let proxy_delegation = self.config.signer_client.generate_proxy_key_bls(pubkey).await?; -let proxy_pubkey = proxy_delegation.message.proxy; - -// or ECDSA proxy -let proxy_delegation = self.config.signer_client.generate_proxy_key_ecdsa(pubkey).await?; -let proxy_address = proxy_delegation.message.proxy; -``` - -Where `pubkey` is the validator (consensus) public key for which a proxy is to be generated. - -Then you can use the generated proxy key to request a signature: -```rust -// if `proxy_pubkey` is a BLS proxy -let datagram = Datagram { data: 1 }; -let request = SignProxyRequest::builder(proxy_pubkey).with_msg(&datagram); -let signature = config.signer_client.request_proxy_signature_bls(&request).await.unwrap(); - -// or for ECDSA proxy -let datagram = Datagram { data: 1 }; -let request = SignProxyRequest::builder(proxy_address).with_msg(&datagram); -let signature = config.signer_client.request_proxy_signature_ecdsa(&request).await.unwrap(); -``` - -## Metrics -We provide support for modules to record custom metrics which are automatically scraped by Prometheus. This involves three steps -### Define metrics -You can use the `prometheus` crate to create a custom registry and metrics, for example: - -```rust -static ref MY_CUSTOM_REGISTRY: Registry = Registry::new_custom(Some("da_commit".to_string()), None).unwrap(); -static ref SIG_RECEIVED_COUNTER: IntCounter = IntCounter::new("signature_received", "successful signature requests received").unwrap(); -``` - -### Start Metrics Provider -When starting the module, you should register all metrics, and start the `MetricsProvider`: -```rust -MY_CUSTOM_REGISTRY.register(Box::new(SIG_RECEIVED_COUNTER.clone())).unwrap(); -MetricsProvider::load_and_run(MY_CUSTOM_REGISTRY.clone()); -``` -The `MetricsProvider` will load the configuration needed and start a server with a `/metrics` endpoint for Prometheus to scrape. - -### Record metrics -All that is left is to use the metrics throughout your code: -```rust -SIG_RECEIVED_COUNTER.inc(); -``` -These will be automatically scraped by the Prometheus service running, and exposed on port `9090`. We plan to allow developers to ship pre-made dashboards together with their modules, to allow Node Operators to have an improved oversight on the modules they are running. diff --git a/docs/docs/developing/commit-modules.md b/docs/docs/developing/commit-modules.md new file mode 100644 index 000000000..bf01271f8 --- /dev/null +++ b/docs/docs/developing/commit-modules.md @@ -0,0 +1,169 @@ +--- +sidebar_position: 1 +--- + +# Commit Modules + +Commit-Boost provides an open platform for developers to create and distribute commitment protocol sidecars. **Commit Modules** are the primary way to add custom logic: they run as sidecar processes alongside the PBS and Signer services, and can request signatures from the proposer. + +> **For system context**, see the [Architecture Overview](../architecture/overview.md). + +## Config entry + +Each commit module is declared in the `cb-config.toml` file under a `[[modules]]` entry: + +```toml +[[modules]] +id = "DA_COMMIT" +type = "commit" +docker_image = "my-module-image" +signing_id = "0x6a33a23ef26a4836979edff86c493a69b26ccf0b4a16491a815a13787657431b" +``` + +| Field | Description | +|---|---| +| `id` | A unique identifier for the module (used for JWT scoping and container naming). | +| `type` | Must be `"commit"`. | +| `docker_image` | The Docker image to run for this module. | +| `signing_id` | A 32-byte identifier used to scope signatures to this module (see [Signing structure](#signing-structure)). | +| `env` | Optional map of environment variables for the module. | +| `env_file` | Optional path to an environment file for the module. | +| (custom) | Additional fields are passed through as opaque config to the module's runtime. | + +:::warning +Setting `type = "pbs"` in a `[[modules]]` entry is **not** a supported path. The configuration parser will reject it at parse time. If you want to extend the PBS binary itself, see [Extending PBS](./extending-pbs.md). +::: + + + +## Rust SDK usage + +While a module can be written in any language, we provide Rust utilities to simplify loading and running modules. Add to your `Cargo.toml`: + +```toml +commit-boost = { git = "https://github.com/Commit-Boost/commit-boost-client", version = "..." } +``` + +Import the prelude: + +```rust +use commit_boost::prelude::*; +``` + +### Loading module config + +Your module will likely need a configuration section for the Node Operator to customize. Define it as a struct and pass it to `load_commit_module_config`: + +```rust +#[derive(Debug, Deserialize)] +struct ExtraConfig { + sleep_secs: u64, +} + +let mut config = load_commit_module_config::().unwrap(); +let to_sleep = config.extra.sleep_secs; +``` + +The returned `StartCommitModuleConfig` also provides: +- `id`: unique module ID +- `chain`: chain spec +- `signer_client`: a pre-configured `SignerClient` to call the [SignerAPI](/api) + +### Requesting signatures + +At its core, the Signer service provides a signature on a 32-byte data digest. Signatures are provided using either the validator keys (BLS) or a proxy key (BLS or ECDSA), both on the [Commit-Boost domain](#signing-structure). + +Use `TreeHash` to create a digest from a custom struct: + +```rust +#[derive(TreeHash)] +struct Datagram { + data: u64, +} +``` + +To request a signature, you need a public key. `get_pubkeys` returns a `GetPubkeysResponse` whose `keys` field maps each consensus key to its proxy keys; pick the consensus pubkey to sign with: + +```rust +let pubkeys = config.signer_client.get_pubkeys().await.unwrap(); +let pubkey = pubkeys.keys.first().unwrap().consensus.clone(); +``` + +JWT tokens are created and refreshed internally by `SignerClient`: each method generates a fresh token with the correct `route`, `exp`, and `payload_hash` claims automatically. No manual token management is needed. + +#### Consensus key signatures + +```rust +let datagram = Datagram { data: 1 }; +let request = SignConsensusRequest::builder(pubkey.clone()).with_msg(&datagram); +let response = config.signer_client.request_consensus_signature(request).await.unwrap(); +let signature = response.signature; +``` + +The response also carries the `nonce` and `module_signing_id` needed to verify the signature. + +#### Proxy key signatures + +First, generate a proxy key for a given consensus key. We support BLS and ECDSA: + +```rust +// BLS proxy +let proxy_delegation = config.signer_client.generate_proxy_key_bls(pubkey.clone()).await?; +let proxy_pubkey = proxy_delegation.message.proxy; + +// ECDSA proxy +let proxy_delegation = config.signer_client.generate_proxy_key_ecdsa(pubkey.clone()).await?; +let proxy_address = proxy_delegation.message.proxy; +``` + +Then request a signature using the proxy key: + +```rust +// BLS proxy +let datagram = Datagram { data: 1 }; +let request = SignProxyRequest::builder(proxy_pubkey).with_msg(&datagram); +let response = config.signer_client.request_proxy_signature_bls(request).await.unwrap(); +let signature = response.signature; + +// ECDSA proxy +let datagram = Datagram { data: 1 }; +let request = SignProxyRequest::builder(proxy_address).with_msg(&datagram); +let response = config.signer_client.request_proxy_signature_ecdsa(request).await.unwrap(); +let signature = response.signature; +``` + +### Signing structure + +For details on the signing structure, including domain separation, nonces, SSZ Merkle tree construction, and the signing ID format, see [Requesting Proposer Commitment Signatures](./prop-commit-signing.md). + +## Metrics + +Modules can record custom metrics that are automatically scraped by Prometheus. + +### Define metrics + +Use the `prometheus` crate, with the statics wrapped in `lazy_static!` (from the `lazy_static` crate): + +```rust +lazy_static! { + static ref MY_CUSTOM_REGISTRY: Registry = Registry::new_custom(Some("da_commit".to_string()), None).unwrap(); + static ref SIG_RECEIVED_COUNTER: IntCounter = IntCounter::new("signature_received", "successful signature requests received").unwrap(); +} +``` + +### Start the metrics provider + +```rust +MY_CUSTOM_REGISTRY.register(Box::new(SIG_RECEIVED_COUNTER.clone())).unwrap(); +MetricsProvider::load_and_run(config.chain, MY_CUSTOM_REGISTRY.clone()).unwrap(); +``` + +This starts a server with a `/metrics` endpoint on the port from `CB_METRICS_PORT` (see [Running with binary](../get_started/running/binary.md)). + +### Record metrics + +```rust +SIG_RECEIVED_COUNTER.inc(); +``` + +For a full reference of available metrics, see the [Metrics catalog](../get_started/running/metrics-catalog.md). When `[metrics]` is enabled, `commit-boost init` prints the scrape targets for each service; add them to your own Prometheus config (see [Metrics](../get_started/running/metrics.md)). diff --git a/docs/docs/developing/custom-modules.md b/docs/docs/developing/custom-modules.md deleted file mode 100644 index cf2244480..000000000 --- a/docs/docs/developing/custom-modules.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -sidebar_position: 1 ---- - -# Custom Modules - -Commit-Boost aims to provide an open platform for developers to create and distribute commitment protocols sidecars. - -There are two ways to leverage Commit-Boost modularity: - -1. Commit Modules, which request signatures from the proposer, e.g. for preconfirmations ([example](https://github.com/Commit-Boost/commit-boost-client/tree/78bdc47bf89082f4d1ea302f9a3f86f609966b28/examples/da_commit)). -2. PBS Modules, which tweak the default PBS Module with additional logic, e.g. verifying additional constraints in `get_header` ([example](https://github.com/Commit-Boost/commit-boost-client/tree/78bdc47bf89082f4d1ea302f9a3f86f609966b28/examples/status_api)). diff --git a/docs/docs/developing/extending-pbs.md b/docs/docs/developing/extending-pbs.md new file mode 100644 index 000000000..07b787b3b --- /dev/null +++ b/docs/docs/developing/extending-pbs.md @@ -0,0 +1,124 @@ +--- +sidebar_position: 2 +--- + +# Extending PBS + +The PBS binary that ships with Commit-Boost can be extended with custom logic. This is **not** a config-level module declaration like commit modules: you replace the PBS binary entirely by implementing the `BuilderApi` trait (the default implementation is the `DefaultBuilderApi` struct). For system context on how the PBS service fits into the Commit-Boost architecture, see the [Architecture Overview](../architecture/overview.md). + +## Before you extend PBS + +| You want to... | Use... | +|---|---| +| Request signatures from the proposer's validator keys (BLS) or proxy keys (BLS/ECDSA) | [Commit module](./commit-modules.md): runs as a sidecar alongside PBS | +| Add custom constraints to `get_header`, `submit_block`, or other BuilderAPI methods | Extend PBS: implement `BuilderApi` on your own struct | +| Run custom logic that triggers on each slot but does not modify relay interaction | Commit module: cheaper to maintain and deploy independently | +| Add new HTTP routes alongside the standard BuilderAPI | Extend PBS: implement `extra_routes()` on your custom `BuilderApi` | + +## How it works + +The PBS binary ships with the [`DefaultBuilderApi`](https://github.com/Commit-Boost/commit-boost-client/blob/main/crates/pbs/src/api.rs) struct, which implements [`BuilderApi`](https://github.com/Commit-Boost/commit-boost-client/blob/main/crates/pbs/src/api.rs) with default (MEV-Boost-compatible) behavior for each method. + +The trait covers: + +- `get_header`: fetch the best header from relays +- `get_status`: check relay health +- `submit_block`: publish blinded blocks +- `register_validator`: register validators with relays +- `reload`: hot-reload configuration +- `extra_routes`: add custom HTTP endpoints + +PBS serves the submit-block route on both `POST /eth/v1/builder/blinded_blocks` and `POST /eth/v2/builder/blinded_blocks`; both are handled by the same `submit_block` method, which receives an `api_version: BuilderApiVersion` parameter (`V1` or `V2`) telling it which route was called. + +By implementing `BuilderApi` on your own struct, you can override any of these methods while reusing the default MEV-Boost logic: the default handlers (`get_header`, `get_status`, `register_validator`, `submit_block`) are re-exported in `commit_boost::prelude`, so you can call them from within your override: + +```rust +use commit_boost::prelude::*; + +// e.g. inside your `get_status` override +get_status(req_headers, state).await +``` + +The default `reload` handler is not re-exported in the prelude, so a `reload` override must rebuild its state itself. + +### Reference example + +See [`examples/status_api/`](https://github.com/Commit-Boost/commit-boost-client/tree/main/examples/status_api) for a complete working example that: + +1. Defines a custom `ExtraConfig` struct with additional TOML fields (`inc_amount`). +2. Creates a custom `BuilderApiState` (`MyBuilderState`) to hold runtime state. +3. Implements `BuilderApi` that overrides `get_status` with custom logging and a counter, and adds a `/check` route via `extra_routes()`. +4. Loads config with `load_pbs_custom_config::()` and starts the service with `PbsService::run::(state)`. + +## Building and running a custom PBS binary + +### Dependencies + +Add the `commit-boost` crate to your `Cargo.toml`: + +```toml +commit-boost = { git = "https://github.com/Commit-Boost/commit-boost-client", version = "..." } +``` + +The snippets below also need `serde` (with the `derive` feature) and `async-trait` in `Cargo.toml`, since the prelude does not re-export them; import them with `use serde::Deserialize;` and `use async_trait::async_trait;`. + +### Entry point + +Your `main.rs` should: + +1. Define your extra config (if any): + +```rust +#[derive(Debug, Deserialize)] +struct ExtraConfig { + inc_amount: u64, +} +``` + +2. Define your state (if any): + +```rust +#[derive(Clone)] +struct MyBuilderState { /* ... */ } +impl BuilderApiState for MyBuilderState {} +``` + +3. Implement `BuilderApi`: + +```rust +struct MyBuilderApi; + +#[async_trait] +impl BuilderApi for MyBuilderApi { + // Override methods here +} +``` + +4. Load config and run: + +```rust +use std::path::PathBuf; + +let (pbs_config, extra) = load_pbs_custom_config::().await?; + +// The second argument is the path PBS watches for config hot-reloads. +// An empty path disables the watcher; see the note below. +let config_path = PathBuf::new(); + +let state = PbsState::new(pbs_config, config_path).with_data(MyBuilderState::from_config(extra)); +PbsService::run::(state).await +``` + +:::note Config hot-reload is opt-in for custom binaries + +`PbsService::run` only spawns the config file watcher when the `config_path` handed to +`PbsState::new` is a non-empty path. `examples/status_api` passes `PathBuf::new()`, so that example +does **not** hot-reload: config changes need a restart. + +To get the same auto-reload behavior as the stock PBS binary, pass the real path of your config file +(typically the value of `CB_CONFIG`) instead of an empty `PathBuf`. +::: + +### Running + +Compile and run your binary. Set the same environment variables as the default PBS (see [Running with binary](../get_started/running/binary.md)). Your custom PBS handles the same BuilderAPI endpoints plus any extra routes you added. diff --git a/docs/docs/developing/prop-commit-signing.md b/docs/docs/developing/prop-commit-signing.md index 30f70413a..44b5792c9 100644 --- a/docs/docs/developing/prop-commit-signing.md +++ b/docs/docs/developing/prop-commit-signing.md @@ -1,76 +1,187 @@ -# Requesting Proposer Commitment Signatures with Commit-Boost +# Requesting proposer commitment signatures with Commit-Boost When you create a new validator on the Ethereum network, one of the steps is the generation of a new BLS private key (commonly known as the "validator key" or the "signer key") and its corresponding BLS public key (the "validator pubkey", used as an identifier). Typically this private key will be used by an Ethereum consensus client to sign things such as attestations and blocks for publication on the Beacon chain. These signatures prove that you, as the owner of that private key, approve of the data being signed. However, as general-purpose private keys, they can also be used to sign *other* arbitrary messages not destined for the Beacon chain. -Commit-Boost takes advantage of this by offering a standard known as **proposer commitments**. These are arbitrary messages (albeit with some important rules), similar to the kind used on the Beacon chain, that have been signed by one of the owner's private keys. Modules interested in leveraging Commit-Boost's proposer commitments can construct their own data in whatever format they like and request that Commit-Boost's **signer service** generate a signature for it with a particular private key. The module can then use that signature to verify the data was signed by that user. +Commit-Boost takes advantage of this by offering a standard known as **proposer commitments**. These are arbitrary messages (albeit with some important rules), similar to the kind used on the Beacon chain, that have been signed by one of the owner's private keys. Modules that use Commit-Boost's proposer commitments can construct their own data in whatever format they like and request that Commit-Boost's **Signer service** generate a signature for it with a particular private key. The module can then use that signature to verify the data was signed by that user. Commit-Boost supports proposer commitment signatures for both BLS private keys (identified by their public key) and ECDSA private keys (identified by their Ethereum address). -## Rules of Proposer Commitment Signatures +## Rules of proposer commitment signatures -Proposer commitment signatures produced by Commit-Boost's signer service conform to the following rules: +Proposer commitment signatures produced by Commit-Boost's Signer service conform to the following rules: - Signatures are **unique** to a given EVM chain (identified by its [chain ID](https://chainlist.org/)). Signatures generated for one chain will not work on a different chain. -- Signatures are **unique** to Commit-Boost proposer commitments. The signer service **cannot** be used to create signatures that could be used for other applications, such as for attestations on the Beacon chain. While the signer service has access to the same validator private keys used to attest on the Beacon chain, it cannot create signatures that would get you slashed on the Beacon chain. -- Signatures are **unique** to a particular module. One module cannot, for example, request an identical payload as another module and effectively "forge" a signature for the second module; identical payloads from two separate modules will result in two separate signatures. -- The data payload being signed must be a **32-byte array**, typically serializd as a 64-character hex string with an optional `0x` prefix. The value itself is arbitrary, as long as it has meaning to the requester - though it is typically the 256-bit hash of some kind of data. +- Signatures are **unique** to Commit-Boost proposer commitments. The Signer service **cannot** be used to create signatures that could be used for other applications, such as for attestations on the Beacon chain. While the Signer service has access to the same validator private keys used to attest on the Beacon chain, it cannot create signatures that would get you slashed on the Beacon chain. +- Signatures are **unique** to a particular module; identical payloads from two modules produce two different signatures (see [The signing ID](#the-signing-id)). +- The data payload being signed must be a **32-byte array**, typically serialized as a 64-character hex string with an optional `0x` prefix. The value itself is arbitrary, as long as it has meaning to the requester, though it is typically the 256-bit hash of some kind of data. - If requesting a signature from a BLS key, the resulting signature will be a standard BLS signature (96 bytes in length). - If requesting a signature from an ECDSA key, the resulting signature will be a standard Ethereum RSV signature (65 bytes in length). -- Signatures **may** be **unique** per request, using the optional `nonce` field in their requests to indicate a unique sequence that this signature belongs to. +- The `nonce` field can make signatures unique per request (see [Nonces](#nonces)). -## Configuring a Module for Proposer Commitments +## Configuring a module for proposer commitments -Commit-Boost's signer service must be configured prior to launching to expect requests from your module. There are two main parts: +Commit-Boost's Signer service must be configured prior to launching to expect requests from your module. There are two main parts: 1. An entry for your module into [Commit-Boost's configuration file](../get_started/configuration.md#custom-module). This must include a unique ID for your module, the line `type = "commit"`, and include a unique [signing ID](#the-signing-id) for your module. Generally you should provide values for these in your documentation, so your users can reference it when configuring their own Commit-Boost node. -2. A JWT secret used by your module to authenticate with the signer in HTTP requests. This must be a string that both the Commit-Boost signer can read and your module can read, but no other modules should be allowed to access it. The user should be responsible for determining an appropriate secret and providing it to the Commit-Boost signer service securely; your module will need some way to accept this, typically via a command line argument that accepts a path to a file with the secret or as an environment variable. +2. A JWT secret used by your module to authenticate with the signer in HTTP requests. This must be a string that both the Commit-Boost signer can read and your module can read, but no other modules should be allowed to access it. The user should be responsible for determining an appropriate secret and providing it to the Commit-Boost Signer service securely; your module will need some way to accept this, typically via a command line argument that accepts a path to a file with the secret or as an environment variable. -Once the user has configured both Commit-Boost and your module with these settings, your module will be able to authenticate with the signer service and request signatures. +Once the user has configured both Commit-Boost and your module with these settings, your module will be able to authenticate with the Signer service and request signatures. -## The Signing ID +## The signing ID Your module's signing ID is a 32-byte value that is used as a unique identifier within the signing process. Proposer commitment signatures incorporate this value along with the data being signed as a way to create signatures that are exclusive to your module, so other modules can't maliciously construct signatures that appear to be from your module. Your module must have this ID incorporated into itself ahead of time, and the user must include this same ID within their Commit-Boost configuration file section for your module. Commit-Boost does not maintain a global registry of signing IDs, so this is a value you should provide to your users in your documentation. -The Signing ID is decoupled from your module's human-readable name (the `module_id` field in the Commit-Boost configuration file) so that any changes to your module name will not invalidate signatures from previous versions. Similarly, if you don't change the module ID but *want* to invalidate previous signatures, you can modify the signing ID and it will do so. Just ensure your users are made aware of the change, so they can update it in their Commit-Boost configuration files accordingly. +The Signing ID is decoupled from your module's human-readable name (the `id` field of the `[[modules]]` entry in the Commit-Boost configuration file) so that any changes to your module name will not invalidate signatures from previous versions. Similarly, if you don't change the module ID but *want* to invalidate previous signatures, you can modify the signing ID and it will do so. Just ensure your users are made aware of the change, so they can update it in their Commit-Boost configuration files accordingly. ## Nonces -Your module has the option of using **Nonces** for each of its signature requests. Nonces are intended to be unique values that establish a sequence of signature requests, distinguishing one signature from another - even if all of their other payload information is identical. When making a request for a signature, you may include a unique nonce as part of the request; the signature will include it in its data, ensuring that things like replay attacks cannot be used for that signature. +Your module has the option of using **Nonces** for each of its signature requests. Nonces are intended to be unique values that establish a sequence of signature requests, distinguishing one signature from another, even if all of their other payload information is identical. When making a request for a signature, you may include a unique nonce as part of the request; the signature will include it in its data, ensuring that things like replay attacks cannot be used for that signature. -If you want to use them within your module, your module (or whatever remote backend system it connects to) **will be responsible** for storing, comparing, validating, and otherwise using the nonces. Commit-Boost's signer service by itself **does not** store nonces or track which ones have already been used by a given module. +If you want to use them within your module, your module (or whatever remote backend system it connects to) **will be responsible** for storing, comparing, validating, and otherwise using the nonces. Commit-Boost's Signer service by itself **does not** store nonces or track which ones have already been used by a given module. -In terms of implementation, the nonce format conforms to the specification in [EIP-2681](https://eips.ethereum.org/EIPS/eip-2681). It is an unsigned 64-bit integer, with a minimum value of 0 and a maximum value of `2^64-2`. The field is required and is always mixed into the signing root. Modules that do not use nonces for replay protection should always send `0`; modules that do should use a monotonically increasing value per key. +In terms of implementation, the nonce is an unsigned 64-bit integer. Per the convention in [EIP-2681](https://eips.ethereum.org/EIPS/eip-2681) the maximum value is `2^64-2`, though the Signer service does not enforce this cap. The field is required and is always mixed into the signing root. Modules that do not use nonces for replay protection should always send `0`; modules that do should use a monotonically increasing value per key. -## Structure of a Signature +## Structure of a signature -The form proposer commitment signatures take depends on the type of signature being requested. BLS signatures take the [standard form](https://eth2book.info/latest/part2/building_blocks/signatures/) (96-byte values). ECDSA (Ethereum EL) signatures take the [standard Ethereum ECDSA `r,s,v` signature form](https://forum.openzeppelin.com/t/sign-it-like-you-mean-it-creating-and-verifying-ethereum-signatures/697). In both cases, the data being signed is a 32-byte hash - the root hash of a composite two-stage [SSZ Merkle tree](https://thogiti.github.io/2024/05/02/Merkleization.html), described below: +The form proposer commitment signatures take depends on the type of signature being requested. BLS signatures take the [standard form](https://eth2book.info/latest/part2/building_blocks/signatures/) (96-byte values). ECDSA (Ethereum EL) signatures take the [standard Ethereum ECDSA `r,s,v` signature form](https://forum.openzeppelin.com/t/sign-it-like-you-mean-it-creating-and-verifying-ethereum-signatures/697). In both cases, the data being signed is a 32-byte hash: the root hash of a composite two-stage [SSZ Merkle tree](https://thogiti.github.io/2024/05/02/Merkleization.html), described below: -
+![signature structure](../res/img/prop_commit_tree.png) - - -
where, for the sub-tree in blue: - `Request Data` is a 32-byte array that serves as the data you want to sign. This is typically a hash of some more complex data on its own that your module constructs. -- `Signing ID` is your module's 32-byte signing ID. The signer service will load this for your module from its configuration file. +- `Signing ID` is your module's 32-byte signing ID. The Signer service will load this for your module from its configuration file. -- `Nonce` is the nonce value for the signature request. This field is required. Modules that do not use replay protection should always send `0`; modules that do should use a monotonically increasing value per key. Conforming with the tree specification, it must be added as a 256-bit unsigned little-endian integer. Most libraries will be able to do this conversion automatically if you specify the field as the language's primitive for 64-bit unsigned integers (e.g., `uint64`, `u64`, `ulong`, etc.). +- `Nonce` is the request's nonce (see [Nonces](#nonces)). Conforming with the tree specification, it must be added as a 256-bit unsigned little-endian integer. Most libraries will be able to do this conversion automatically if you specify the field as the language's primitive for 64-bit unsigned integers (e.g., `uint64`, `u64`, `ulong`, etc.). - `Chain ID` is the ID of the chain that the Signer service is currently configured to use, as indicated by the [Commit-Boost configuration file](../get_started/configuration.md). This must also be a 256-bit unsigned little-endian integer. A Merkle tree must be constructed from these four leaf nodes, and its root hash calculated according to the standard SSZ hash computation rules. This result will be called the "sub-tree root". With this, a second Merkle tree is created using this sub-tree root and a value called the Domain: -- `Domain` is the 32-byte output of the [compute_domain()](https://eth2book.info/capella/part2/building_blocks/signatures/#domain-separation-and-forks) function in the Beacon specification. The 4-byte domain type in this case is not a standard Beacon domain type, but rather Commit Boost's own domain type: `0x6D6D6F43`. +- `Domain` is the 32-byte output of the [compute_domain()](https://eth2book.info/capella/part2/building_blocks/signatures/#domain-separation-and-forks) function in the Beacon specification. The 4-byte domain type in this case is not a standard Beacon domain type, but rather Commit-Boost's own domain type: `0x6D6D6F43`. The data signed in a proposer commitment is the 32-byte hash root of this new tree (the green `Root` box). Many languages provide libraries for computing the root of an SSZ Merkle tree, such as [fastssz for Go](https://github.com/ferranbt/fastssz) or [tree_hash for Rust](https://docs.rs/tree_hash/latest/tree_hash/). When verifying proposer commitment signatures, use a library that supports Merkle tree root hashing, the `compute_domain()` operation, and validation for signatures generated by your key of choice. + +--- + +## Authentication + +Every request to the Signer service (except the health-check endpoint) must present a Bearer token in the `Authorization` header. + +### Module JWT + +Modules authenticate with a **signed JWT** using the pre-shared secret (`CB_SIGNER_JWT` env var). The JWT is an HS256 token with the following claims: + +| Claim | Type | Required | Description | +|-------|------|----------|-------------| +| `module` | string | always | The module's `id` from the `[[modules]]` entry in `cb-config.toml`. | +| `route` | string | always | The exact request path, e.g. `/signer/v1/get_pubkeys`. | +| `exp` | integer | always | UNIX timestamp for when the token expires. | +| `payload_hash` | string | POST only | Keccak-256 hash of the JSON-encoded request body, with `0x` prefix. Skipped for GET requests. | + +The `payload_hash` claim prevents JWT replay attacks: a token issued for one POST request body cannot be reused with a different body on the same route. + +**Token lifecycle:** Expiry is 5 minutes (`SIGNER_JWT_EXPIRATION` crate constant). Refresh is **client-side**: there is no refresh endpoint. The module generates a new JWT locally using the pre-shared secret. The SDK's `SignerClient` creates a fresh token on every request automatically. + +### Admin token + +Administrative endpoints (`/reload`, `/revoke_jwt`) authenticate with a **separate JWT** signed with the `CB_SIGNER_ADMIN_JWT` secret (env var), using the same HS256 algorithm. Its claims are the module claims minus `module`, plus an admin flag: + +| Claim | Type | Required | Description | +|-------|------|----------|-------------| +| `admin` | boolean | always | Must be `true`. | +| `route` | string | always | The exact request path, `/reload` or `/revoke_jwt`. | +| `exp` | integer | always | UNIX timestamp for when the token expires. | +| `payload_hash` | string | always in practice | Keccak-256 hash of the JSON-encoded request body, with `0x` prefix. Both admin endpoints require a JSON body, so every admin request needs this claim. | + +### Rate limiting + +The Signer service rate-limits failed authentications by client IP (default 3 per 5 minutes); limits and reverse-proxy IP extraction are configured in `[signer]`, see [Configuration > Rate limit](../get_started/configuration.md#rate-limit). + +--- + +## API quickstart + +Below is a walkthrough of the full Signer API flow using the Rust SDK. The `SignerClient` (returned by `load_commit_module_config`) handles token management for you (see [Module JWT](#module-jwt)). Besides `commit-boost`, your `Cargo.toml` needs `serde` with the `derive` feature for the config struct, `tokio` with the `macros` and `rt-multi-thread` features for the async runtime, and `eyre` for the error type. + +```rust +use commit_boost::prelude::*; +use serde::Deserialize; + +// 1. Load the module config; this gives you a pre-configured SignerClient +#[derive(Debug, Deserialize)] +struct ExtraConfig { /* your module's custom fields */ } + +#[tokio::main] +async fn main() -> eyre::Result<()> { + let config = load_commit_module_config::()?; + let mut client = config.signer_client; + + // 2. List available validator pubkeys + let pubkeys = client.get_pubkeys().await?; + println!("Loaded {} validators", pubkeys.keys.len()); + + // 3. Generate a BLS proxy key for a consensus pubkey + let consensus = pubkeys.keys[0].consensus.clone(); + let delegation = client.generate_proxy_key_bls(consensus.clone()).await?; + let proxy_pubkey = delegation.message.proxy; + + // 4. Request a signature with the consensus key + #[derive(TreeHash)] + struct Datagram { data: u64 } + + let datagram = Datagram { data: 42 }; + let request = SignConsensusRequest::builder(consensus).with_msg(&datagram); + let sig = client.request_consensus_signature(request).await?; + + // 5. Or request a signature with the proxy key + let proxy_request = SignProxyRequest::builder(proxy_pubkey).with_msg(&datagram); + let proxy_sig = client.request_proxy_signature_bls(proxy_request).await?; + + Ok(()) +} +``` + +For a complete working example, see [`examples/da_commit/`](https://github.com/Commit-Boost/commit-boost-client/tree/main/examples/da_commit) in the repository. + +--- + +## Common workflows + +### Requesting a BLS consensus signature +![Requesting a BLS consensus signature](../res/img/consensus-key-sign.png) + +### Generating and using a proxy key +![Generating and using a proxy key](../res/img/proxy-key-sign.png) + +The proxy private key never leaves the signer, despite the diagram's "store proxy key securely" note. The module receives only the `SignedProxyDelegation` (the delegator and proxy identifiers plus the delegation signature) and stores that, not the key. + +:::tip ECDSA proxy signing with Dirk +ECDSA proxy signing is not available when the signer is using the Dirk backend. Dirk only supports BLS operations. +::: + +--- + +## Error codes + +All error responses return a plain-text body with a human-readable description of the error. + +| HTTP Status | Meaning | +|-------------|---------| +| `400` | Missing or malformed `Authorization` header, unreadable or oversized request body, missing signing ID, or operation not supported by the current backend (e.g. ECDSA proxy with Dirk). | +| `401` | Invalid JWT: expired, signed with the wrong secret, or claims that do not match the request. A missing or malformed `Authorization` header returns `400` instead. | +| `404` | Requested consensus signer, proxy signer, or module ID does not exist. | +| `422` | Request body failed deserialization, e.g. a pubkey that is not valid hex of the expected length. | +| `429` | Too many failed authentication attempts. Retry after the timeout period. | +| `500` | Internal server error. The request was valid but could not be fulfilled. | +| `502` | Signer is running in Dirk mode but Dirk is unreachable. | diff --git a/docs/docs/get_started/building.md b/docs/docs/get_started/building.md index 1b78edf35..8b84340d3 100644 --- a/docs/docs/get_started/building.md +++ b/docs/docs/get_started/building.md @@ -1,33 +1,55 @@ # Building Commit-Boost from Source -Commit-Boost's components are all written in [Rust](https://www.rust-lang.org/). This guide will walk you through the setup required to build them from source. It assumes you are on a Debian or Debian-based system (e.g., Ubuntu, Linux Mint, Pop OS). For other systems, please adapt the steps for your system's package manager accordingly. +Commit-Boost's components are all written in [Rust](https://www.rust-lang.org/). This guide walks through building them from source. + +## Getting the Source + +Pull the repository: + +```bash +git clone https://github.com/Commit-Boost/commit-boost-client +``` + +Check out the release you want to build. Each release is pinned in `.releases/` +(e.g. `.releases/v0.10.0.yml` names its `commit:`); check out that commit: + +```bash +cd commit-boost-client && git checkout +``` + +Finally, update the submodules: + +``` +git submodule update --init --recursive +``` ## Building via the Docker Builder -For convenience, Commit-Boost has Dockerized the build environment for Linux `x64` and `arm64` platforms. It utilizes Docker's powerful [buildx](https://docs.docker.com/reference/cli/docker/buildx/) system. All of the prerequisites, cross-compilation tooling, and configuration are handled by the builder image. If you would like to build the Commit-Boost binary and Docker image from source, you are welcome to use the Docker builder process. +The build environment is Dockerized for the Linux `x64` and `arm64` platforms, using Docker's [buildx](https://docs.docker.com/reference/cli/docker/buildx/) system. The builder image handles all of the prerequisites, cross-compilation tooling, and configuration, so this path does not need a local Rust toolchain. -To use the builder, you will need to have [Docker Engine](https://docs.docker.com/engine/install/) installed on your system. Please follow the instructions to install it first. +The builder requires [Docker Engine](https://docs.docker.com/engine/install/). :::note The build system assumes that you've added your user account to the `docker` group with the Linux [post-install steps](https://docs.docker.com/engine/install/linux-postinstall/). If you haven't, then you'll need to run the build script below as `root` or modify it so each call to `docker` within it is run as the root user (e.g., with `sudo`). ::: -The Docker builder is built into the project's `justfile` which is used to invoke many facets of Commit Boost development. To use it, you'll need to install [Just](https://github.com/casey/just) on your system. - -Use `just --list` to show all of the actions - there are many. The `justfile` provides granular actions, called "recipes", for building just the binaries of a specific crate (such as the CLI, `pbs`, or `signer`), as well as actions to build the Docker images for the PBS and Signer services. +Builds run through the project's `justfile` (install [Just](https://github.com/casey/just); `just --list` shows all recipes). The relevant recipes: `build-bin ` builds the `commit-boost` binary; `build-all ` additionally builds the unified Docker image `commit-boost/commit-boost:` (bundling all subcommands) and loads it into your local registry. `` is the output directory name under `./build/` and the Docker tag, e.g. `$(git rev-parse --short HEAD)`. For Linux `amd64` + `arm64` use the `-multiarch` recipe variants; a multiarch image manifest needs a [custom Docker registry](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-ubuntu-20-04), as Docker's built-in local registry does not support them. -Below is a brief summary of the relevant ones for building the Commit-Boost artifacts: +To build the binary, run: -- `build-all ` builds the `commit-boost` binary to `./build/` and creates a Docker image called `commit-boost/commit-boost:` (a unified image that bundles all subcommands), loading it into your local Docker registry. -- `build-bin ` can be used to create the `commit-boost` binary itself. +``` +just build-bin +``` -The `version` provided will be used to house the output binaries in `./build/`, and act as the version tag for the Docker images when they're added to your local system or uploaded to your local Docker repository. For example, using `$(git rev-parse --short HEAD)` will set the version to the current commit hash. +This will create a binary in `build//`, for example `build/206658b/linux_amd64/`. Confirm that it works: -If you're interested in building the binaries and/or Docker images for multiple architectures (currently Linux `amd64` and `arm64`), use the variants of those recipes that have the `-multiarch` suffix. Note that building a multiarch Docker image manifest will require the use of a [custom Docker registry](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-ubuntu-20-04), as the local registry built into Docker does not have multiarch manifest support. +``` +./build///commit-boost --version +``` ## Building Manually -If you don't want to use the Docker builder, you can compile the Commit-Boost artifacts locally. The following instructions assume a Debian or Debian-based system (e.g., Ubuntu, Linux Mint, Pop OS) for simplicity. For other systems, please adapt any relevant instructions to your environment accordingly. +If you don't want to use the Docker builder, you can compile the Commit-Boost artifacts with a local Rust toolchain. The following instructions assume a Debian or Debian-based system (e.g., Ubuntu, Linux Mint, Pop OS) for simplicity. For other systems, please adapt any relevant instructions to your environment accordingly. ### Prerequisites @@ -38,7 +60,7 @@ Requirements: - OpenSSL development libraries - Protobuf Compiler (`protoc`) -Start by installing Rust if you don't already have it. Follow [the official directions](https://www.rust-lang.org/learn/get-started) to install it and bring it up to date. +Install Rust via [the official directions](https://www.rust-lang.org/learn/get-started) if you don't already have it. Install the dependencies: @@ -52,53 +74,37 @@ Install the Protobuf compiler: While many package repositories provide a `protobuf-compiler` package in lieu of manually installing protoc, we've found at the time of this writing that Debian-based ones use v3.21 which is quite out of date. We recommend getting the latest version manually. ::: -We provide a convenient recipe to install the latest version directly from the GitHub releases page: +The repository provides a recipe to install the latest version directly from the GitHub releases page. It requires [Just](https://github.com/casey/just) and works from anywhere inside the repository: ```bash just install-protoc ``` -This works on OSX and Linux systems, but you are welcome to download and install it manually as well. - -With the prerequisites set up, pull the repository: - -```bash -git clone https://github.com/Commit-Boost/commit-boost-client -``` - -Check out the `stable` branch which houses the latest release: - -```bash -cd commit-boost-client && git checkout stable -``` - -Finally, update the submodules: - -``` -git submodule update --init --recursive -``` +This works on OSX and Linux systems. You can also run `provisioning/protoc.sh` directly, or download and install protoc manually. Your build environment should now be ready to use. ### Building the Binary -To build the binary, run: +From the repository root, build the unified `commit-boost` binary with Cargo: ``` -just build-bin +cargo build --release --bin commit-boost ``` -This will create a binary in `build//`, for example `build/206658b/linux_amd64/`. Confirm that it works: +This will create the binary at `target/release/commit-boost`. Confirm that it works: ``` -./build///commit-boost --version +./target/release/commit-boost --version ``` You can now use this to generate the Docker Compose file to drive the other modules if desired. See the [configuration](./configuration.md) guide for more information. ### Verifying the PBS Service -To verify the PBS service works, create [a TOML configuration](./configuration.md) for the PBS module (e.g., `cb-config.toml`). +The commands below use the manual build's output path (`./target/release/commit-boost`); if you used the Docker builder, substitute `./build///commit-boost`. + +To verify the PBS service works, create [a TOML configuration](./configuration.md) for the PBS service (e.g., `cb-config.toml`). As a quick example, we'll use this configuration that connects to the Flashbots relay on the Hoodi network: @@ -107,7 +113,6 @@ chain = "Hoodi" [pbs] port = 18550 -with_signer = true [[relays]] url = "https://0xafa4c6985aa049fb79dd37010438cfebeb0f2bd42b115b89dd678dab0670c1de38da0c4e9138c9290a398ecd9a0b3110@boost-relay-hoodi.flashbots.net" @@ -122,47 +127,52 @@ port = 20000 format = "lighthouse" keys_path = "/tmp/keys" secrets_path = "/tmp/secrets" + +[[modules]] +id = "test" +type = "commit" +docker_image = "test_module" +signing_id = "0x6a33a23ef26a4836979edff86c493a69b26ccf0b4a16491a815a13787657431b" ``` Set the path to it in the `CB_CONFIG` environment variable and run the binary: ``` -CB_CONFIG=cb-config.toml ./build///commit-boost pbs +CB_CONFIG=cb-config.toml ./target/release/commit-boost pbs ``` If it works, you should see output like this: ``` -2025-05-07T21:09:17.407245Z WARN No metrics server configured -2025-05-07T21:09:17.407257Z INFO starting PBS service version="0.7.0" commit_hash="58082edb1213596667afe8c3950cd997ab85f4f3" addr=127.0.0.1:18550 events_subs=0 chain=Hoodi -2025-05-07T21:09:17.746855Z INFO : new request ua="" relay_check=true method=/eth/v1/builder/status req_id=5c405c33-0496-42ea-a35d-a7a01dbba356 -2025-05-07T21:09:17.896196Z INFO : relay check successful method=/eth/v1/builder/status req_id=5c405c33-0496-42ea-a35d-a7a01dbba356 +2025-11-04T14:22:03.118512Z INFO starting PBS service version="0.10.0-rc4" commit_hash="eeff25750c01f4adfc95fc08d69d541ace8e4087" addr=127.0.0.1:18550 chain=Hoodi ``` -If you do, then the PBS service works. +The v0.10.0 release commit self-reports version `0.10.0-rc4`; timestamps will differ, and any other checkout prints its own commit hash and version. A successful relay check follows the `starting PBS service` line; the full annotated healthy-log reference is in [Expected healthy logs](./troubleshooting.md#expected-healthy-logs). -### Verifying the Signer Module +If you see that, then the PBS service works. -To verify the Signer service works, create [a TOML configuration](./configuration.md) for the Signer module (e.g., `cb-config.toml`). We'll use the example in the PBS section above. +### Verifying the Signer Service -The signer needs the following environment variables set: +To verify the Signer service works, create [a TOML configuration](./configuration.md) for the Signer service (e.g., `cb-config.toml`). We'll use the example in the PBS section above. -- `CB_CONFIG` = path of your config file. -- `CB_JWTS` = a dummy key-value pair of [JWT](https://en.wikipedia.org/wiki/JSON_Web_Token) values for various services. Since we don't need them for the sake of just testing the binary, we can use something like `"test_jwts=dummy"`. +The signer needs `CB_CONFIG`, `CB_JWTS`, and `CB_SIGNER_ADMIN_JWT` set (definitions: [Binary > Signer Service](./running/binary.md#signer-service)); for this smoke test use `test=dummy` and a dummy admin secret. Set these values, create the `keys` and `secrets` directories listed in the configuration file, and run the binary: ``` mkdir -p /tmp/keys && mkdir -p /tmp/secrets -CB_CONFIG=cb-config.toml CB_JWTS="test_jwts=dummy" ./build///commit-boost signer +CB_CONFIG=cb-config.toml CB_JWTS="test=dummy" CB_SIGNER_ADMIN_JWT="dummy_admin" ./target/release/commit-boost signer ``` You should see output like this: ``` -2025-06-03T04:57:19.815702Z WARN Proxy store not configured. Proxies keys and delegations will not be persisted -2025-06-03T04:57:19.818193Z INFO Starting signing service version="0.8.0-rc.1" commit_hash="3eed5268f07803c55cca7d7e2e14a7017098f797" modules=["test"] endpoint=127.0.0.1:20000 loaded_consensus=0 loaded_proxies=0 -2025-06-03T04:57:19.818229Z WARN No metrics server configured +2025-11-04T14:31:44.815702Z WARN Proxy store not configured. Proxies keys and delegations will not be persisted +2025-11-04T14:31:44.818193Z INFO Starting signing service version="0.10.0-rc4" commit_hash="eeff25750c01f4adfc95fc08d69d541ace8e4087" modules=["test"] endpoint=127.0.0.1:20000 loaded_consensus=0 loaded_proxies=0 jwt_auth_fail_limit=3 jwt_auth_fail_timeout=300s reverse_proxy=None +2025-11-04T14:31:44.818229Z WARN No metrics server configured +2025-11-04T14:31:44.818305Z WARN Running in insecure HTTP mode, no TLS certificates provided ``` -If you do, then the binary works. +The `insecure HTTP mode` warning is expected: this config does not set a `tls_mode`; see [TLS](./configuration.md#tls) to enable it. + +If you see that, then the binary works. diff --git a/docs/docs/get_started/configuration.md b/docs/docs/get_started/configuration.md index 7eefb2774..929e9f54f 100644 --- a/docs/docs/get_started/configuration.md +++ b/docs/docs/get_started/configuration.md @@ -6,42 +6,141 @@ description: Configure Commit-Boost Commit-Boost needs a configuration file detailing all the services that you want to run. Create a `cb-config.toml` and modify it depending on which modules you plan to run. -- For a full explanation of all the fields, check out [here](https://github.com/Commit-Boost/commit-boost-client/blob/main/config.example.toml). -- For some additional examples on config presets, check out [here](https://github.com/Commit-Boost/commit-boost-client/tree/main/configs). +- For a full explanation of all the fields, see the [annotated config example](https://github.com/Commit-Boost/commit-boost-client/blob/main/config.example.toml). +- For some additional examples, see the [example config presets](https://github.com/Commit-Boost/commit-boost-client/tree/main/examples/configs). -## Minimal PBS setup on Holesky +## Minimal PBS setup on Hoodi ```toml -chain = "Holesky" +chain = "Hoodi" [pbs] port = 18550 [[relays]] -url = "" +# Replace this with your relay's own URL, in the form scheme://@ +url = "https://0xa1cec75a3f0661e99299274182938151e8433c61a19222347ea1313d839229cb4ce4e3e5aa2bdeb71c8fcf1b084963c2@relay.example.com" [metrics] enabled = true ``` -You can find a list of MEV-Boost Holesky relays [here](https://www.coincashew.com/coins/overview-eth/mev-boost/mev-relay-list#holesky-testnet-relays). -After the sidecar is started, it will expose a port (`18550` in this example), that you need to point your CL to. This may be different depending on which CL you're running, check out [here](https://docs.flashbots.net/flashbots-mev-boost/getting-started/system-requirements#consensus-client-configuration-guides) for a list of configuration guides. +:::warning +`url` must be a full URL containing the relay's BLS pubkey; an empty URL or one without a pubkey (`invalid BLS pubkey`) prevents the sidecar from starting. +::: + +You can find a list of MEV-Boost Hoodi relays [here](https://github.com/ethstaker/ethstaker-guides/blob/main/MEV-relay-list.md#mev-relay-list-for-hoodi-testnet). +After the sidecar is started, it will expose a port (`18550` in this example), that you need to point your CL to. This may be different depending on which CL you're running; see the [consensus client configuration guides](https://docs.flashbots.net/flashbots-mev-boost/getting-started/system-requirements#consensus-client-configuration-guides). :::note -In this setup, the signer module will not be started. +In this setup, the Signer service will not be started. ::: -## Signer module +## Custom chains + +Besides the known chain names (`Mainnet`, `Holesky`, `Sepolia`, `Hoodi`), the `chain` field also accepts a custom chain in two forms: + +- **Spec file**: a genesis time plus a path to a chain spec file, either in JSON (as returned by the beacon endpoint `/eth/v1/config/spec`) or YAML format: + +```toml +chain = { genesis_time_secs = 1695902400, path = "/path/to/spec.json" } +``` + +- **Inline object**: all parameters specified directly: + +```toml +chain = { genesis_time_secs = 1695902400, slot_time_secs = 12, genesis_fork_version = "0x01017000", fulu_fork_slot = 5283840, chain_id = 17000 } +``` + +All inline fields are required; omitting one makes the `chain` value fail to parse. + +With the spec-file form, the `CB_CHAIN_SPEC` environment variable can override the spec file path at runtime (see [Binary](./running/binary.md#common)). + +## PBS safety and tuning options + +Beyond the basics shown above, the `[pbs]` section supports additional knobs. The [annotated config example](https://github.com/Commit-Boost/commit-boost-client/blob/main/config.example.toml) is the full field reference with defaults; the options below carry extra operational context: + +- `timeout_get_header_ms`: timeout, in milliseconds, for the `get_header` call to relays. Must be greater than 0, and must be less than `late_in_slot_time_ms`. The CL also has a timeout (e.g. 1 second), so this should be lower than that to leave some margin for overhead. Default: `950`. +- `late_in_slot_time_ms`: how late, in milliseconds into the slot, is considered "late". This shortens `get_header` timeouts to make sure a header is returned within this deadline; if the CL request arrives later in the slot, fetching headers is skipped to force local building and minimize the risk of a missed slot. Must be greater than 0. Default: `2000`. +- `extra_validation_enabled`: whether to enable extra validation of `get_header` responses. If enabled, `rpc_url` must also be set to an Execution Layer RPC on the same chain as the sidecar (this is checked at startup). Default: `false`. +- `mux_registry_refresh_interval_seconds`: refresh interval, in seconds, for registry-based muxes with [dynamic refreshing](./mux-key-loaders.md#lido-registry) enabled. Default: `384`. + +:::warning +`validator_registration_batch_size` used to be a per-relay option. It is no longer accepted per relay: setting it inside a `[[relays]]` entry makes the sidecar fail at startup; set it in the `[pbs]` section instead. +::: + +### Per-relay options + +Each `[[relays]]` entry supports, besides `id` and `url`: + +- `headers`: optional headers to send with each request to this relay. +- `get_params`: optional GET parameters to add to each request URL for this relay. +- `get_header` (unreleased, from v0.11): how headers are fetched from this relay, either `"http"` (one request per `get_header`) or `"stream"` (a websocket stream of bid updates, only for relays that support it; see the annotated config example for the stream endpoint and header handshake). Default: `"http"`. Released v0.10.0 does not recognize this option: setting it in a `[[relays]]` entry fails config parsing at startup. +- `enable_timing_games`: whether to enable timing games for this relay, as tuned by `target_first_request_ms` and `frequency_get_header_ms`. If neither of those is set, this flag has no effect. Advanced users only: misconfiguration can result in e.g. fetching a lower header value or missing a slot (caveats and worked examples in the annotated config example). Default: `false`. +- `target_first_request_ms`: target time in the slot, in milliseconds, at which to send the first `get_header` request. +- `frequency_get_header_ms`: frequency, in milliseconds, at which to send `get_header` requests. + +The same fields are available on `[[mux.relays]]` entries (see [Mux key loaders](./mux-key-loaders.md)). + +#### Header streaming + +:::info Unreleased +This describes behavior on main, unreleased, targeted for v0.11. With `get_header = "stream"`, PBS opens one websocket connection per `get_header` call, keeps the latest bid received until the deadline, then validates and returns it; the timing-game options do not apply while streaming. Any configured `headers` (e.g. an API key) are sent on the websocket handshake. If the connection cannot be established, PBS falls back to a plain HTTP `get_header` with the remaining timeout; a handshake timeout instead surfaces as status `555` in `cb_pbs_relay_status_code_total`, and a relay that rejects the handshake with an HTTP response records that response's own status code. A stream error before any bid arrives yields no header from that relay for the slot, surfaced as `556`; if a bid already arrived, that bid is still returned. +::: + +### SSZ support + +All Builder API requests and responses currently use JSON. + +:::info Unreleased +This describes behavior on main, unreleased, targeted for v0.11: on `get_header` and v1 `submit_blinded_block` requests, PBS negotiates the response encoding with the beacon node through the `Accept` header. Both SSZ and JSON are supported, the response follows the client's `Accept` preference (q-values, then listing order), defaulting to JSON when no preference is expressed, and a request that accepts neither is rejected with `406`. v2 `submit_blinded_block` responses are empty `202`s, so there is nothing to negotiate. Towards relays, PBS always requests SSZ first and falls back to JSON for relays that do not support it. +::: + +## Logs + +Logging is configured via the optional `[logs.stdout]` and `[logs.file]` sections: + +```toml +[logs.stdout] +enabled = true # Whether to enable stdout logging. Default: true +level = "info" # Log level: trace, debug, info, warn, error. Default: "info" +use_json = false # Log in JSON format. Default: false +color = true # Whether to use colors in the output. Default: true + +[logs.file] +enabled = true # Whether to enable file logging. Default: false +level = "info" # Log level: trace, debug, info, warn, error. Default: "info" +use_json = true # Log in JSON format. Default: true +dir_path = "/var/logs/commit-boost" # Directory to store logs. Default: "/var/logs/commit-boost" +max_files = 30 # Maximum number of log files to keep. Default: unlimited +``` + +The `CB_LOGS_DIR` environment variable overrides `dir_path` (see [Binary](./running/binary.md#common)). + +## Metrics + +Prometheus metrics are configured via the optional `[metrics]` section; if the section is missing, metrics collection is disabled: + +```toml +[metrics] +enabled = true # Whether to collect metrics. Default: true +host = "127.0.0.1" # Host to expose the metrics servers on. Default: 127.0.0.1 +start_port = 10000 # First Prometheus scrape port; each service uses start_port, start_port + 1, ... Default: 10000 +``` + +The `CB_METRICS_PORT` environment variable overrides the port used by a module at runtime (see [Binary](./running/binary.md#common)). + +## Signer service -Commit-Boost supports both local and remote signers. The signer module is responsible for signing the transactions that other modules generates. Please note that only one signer at a time is allowed. +Commit-Boost supports both local and remote signers. The Signer service is responsible for signing the transactions that commit modules generate. It is not used by the default PBS image; custom PBS builds can opt in via `pbs.with_signer`. Only one signer at a time is allowed. The config file must still contain at least one `[[relays]]` entry even on a host that runs only the signer; the shared config schema requires it. ### Local signer -To start a local signer module, you need to include its parameters in the config file +To start a local Signer service, you need to include its parameters in the config file: ```toml [pbs] -... +# ... with_signer = true [signer] @@ -50,10 +149,10 @@ port = 20000 [signer.local.loader] format = "lighthouse" keys_path = "/path/to/keys" -secrets_path = "/path/to.secrets" +secrets_path = "/path/to/secrets" ``` -We currently support Lighthouse, Prysm, Teku, Lodestar, and Nimbus's keystores so it's easier to load the keys. We're working on adding support for additional keystores. These are the expected file structures for each format: +Supported keystore formats: Lighthouse, Prysm, Teku, Lodestar, and Nimbus. Expected file structures for each format:
Lighthouse @@ -75,7 +174,7 @@ We currently support Lighthouse, Prysm, Teku, Lodestar, and Nimbus's keystores s ```toml [pbs] -... +# ... with_signer = true [signer] @@ -107,7 +206,7 @@ secrets_path = "secrets" ```toml [pbs] -... +# ... with_signer = true [signer] @@ -139,7 +238,7 @@ secrets_path = "secrets/password.txt" ```toml [pbs] -... +# ... with_signer = true [signer] @@ -170,7 +269,7 @@ secrets_path = "secrets" ```toml [pbs] -... +# ... with_signer = true [signer] @@ -206,7 +305,7 @@ All keys have the same password stored in `secrets/password.txt` #### Config: ```toml [pbs] - ... + # ... with_signer = true [signer] @@ -221,7 +320,7 @@ All keys have the same password stored in `secrets/password.txt` ### Proxy keys store -Proxy keys can be used to sign transactions with a different key than the one used to sign the block. Proxy keys are generated by the Signer module and authorized by the validator key. Each module have their own proxy keys, that can be BLS or ECDSA. +Proxy keys can be used to sign transactions with a different key than the one used to sign the block. Proxy keys are generated by the Signer service and authorized by the validator key. Each module can have its own proxy keys, which can be BLS or ECDSA. To persist proxy keys across restarts, you must enable the proxy store in the config file. There are 2 options for this: @@ -267,7 +366,7 @@ Where each `` file contains the following:
ERC2335 -The keys are stored in a ERC-2335 style keystore, along with a password. This way, you can safely share the keys directory as without the password they are useless. +The keys are stored in an ERC-2335-style keystore, along with a password. This way, you can safely share the keys directory: without the password they are useless. #### File structure @@ -307,20 +406,11 @@ Where the `.json` files contain ERC-2335 keystore, the `///`, so accounts found with that pattern will be ignored. +- `wallets` is a list of wallets from which the Signer service will load all accounts as consensus keys. Generated proxy keys will have format `///`, so accounts found with that pattern will be ignored. - `secrets_path` is the path to the folder containing the passwords of the generated proxy accounts, which will be stored in `////.pass`. Additionally, you can set a proxy store so that the delegation signatures for generated proxy keys are stored locally. As these signatures are not sensitive, the only supported store type is `File`: @@ -354,20 +444,18 @@ Additionally, you can set a proxy store so that the delegation signatures for ge proxy_dir = "/path/to/proxy_dir" ``` -Delegation signatures will be stored in files with the format `/delegations//.sig`. +Delegation signatures will be stored in files with the format `/delegations//bls/.sig`. A full example of a config file with Dirk can be found [here](https://github.com/Commit-Boost/commit-boost-client/blob/main/examples/configs/dirk_signer.toml). ### TLS -By default, the Signer service runs in **insecure** mode, so its API service uses HTTP without any TLS encryption. This is sufficient for testing or if you're running locally within your machine's isolated Docker network and only intend to access it within the confines of your machine. However, for larger production setups, it's recommended to enable TLS - especially for traffic that spans across multiple machines. - -The Signer service in TLS mode supports **TLS 1.2** and **TLS 1.3**. Older protocol versions are not supported. +By default the Signer API uses plain HTTP with no TLS. That is fine for testing or a single-machine Docker network; enable TLS for any traffic that crosses machines. -To enable TLS, you must first create a **certificate / key pair**. We **strongly advise** using a well-known Certificate Authority to create and sign the certificate, such as [Let's Encrypt](https://letsencrypt.org/getting-started/) (a free service) or [Bluehost](https://www.bluehost.com/help/article/how-to-set-up-an-ssl-certificate-for-website-security) (free but requires an account). We do not recommend using a self-signed ceriticate / key pair for production environments. +To enable TLS, you must first create a **certificate / key pair**. We **strongly advise** using a well-known Certificate Authority to create and sign the certificate and do not recommend using a self-signed certificate / key pair for production environments. -When configuring TLS support, the Signer service expects a single folder (which you can specify) that contains the following two files: +When configuring TLS support, the Signer service expects a single folder containing: - `cert.pem`: The SSL certificate file signed by a certificate authority, in PEM format - `key.pem`: The private key corresponding to `cert.pem` that will be used for signing TLS traffic, in PEM format @@ -375,19 +463,19 @@ Specifying it is done within Commit-Boost's configuration file using the `[signe ```toml [pbs] -... +# ... with_signer = true [signer] port = 20000 -... +# ... [signer.tls_mode] type = "certificate" path = "path/to/your/cert/folder" ``` -Where `path` is the aforementioned folder. It defaults to `./certs` but can be replaced with whichever directory your certificate and private key file reside in, as long as they're readable by the Signer service (or its Docker container, if using Docker). +With `type = "certificate"`, an existing `cert.pem` and `key.pem` must be present at `path`; there is no default path and the files are not auto-generated. ### Rate limit @@ -395,15 +483,15 @@ The Signer service implements a rate limit system of 3 failed authentications ev ```toml [signer] -... +# ... jwt_auth_fail_limit = 3 # The amount of failed requests allowed jwt_auth_fail_timeout_seconds = 300 # The time window in seconds ``` -The rate limit is applied to the IP address of the client making the request. By default, the IP is extracted directly from the TCP connection. If you're running the Signer service behind a reverse proxy (e.g. Nginx), you can configure it to extract the IP from a custom HTTP header instead. There're two options: +The rate limit is applied to the IP address of the client making the request. By default, the IP is extracted directly from the TCP connection. If you're running the Signer service behind a reverse proxy (e.g. Nginx), you can configure it to extract the IP from a custom HTTP header instead. There are two options: -- unique: Provides an HTTP header that contains the IP. This header is expected to appear only once in the request. This is common when using `X-Real-IP`, `True-Client-IP`, etc. If a request has multiple values for this header, it will be considered invalid and rejected. -- `rightmost`: Provides an HTTP header that contains a comma-separated list of IPs. The nth rightmost IP in the list is used. If the header appears multiple times, the last occurrence is used. This is common when using `X-Forwarded-For`. +- `unique`: Provides an HTTP header that contains the IP. This header is expected to appear only once in the request. This is common when using `X-Real-IP`, `True-Client-IP`, etc. If a request has multiple values for this header, it will be considered invalid and rejected. +- `rightmost`: Provides an HTTP header that contains a comma-separated list of IPs. The nth rightmost IP in the list is used. If the header appears multiple times, all occurrences are concatenated in order and the rightmost counting spans the combined list. This is common when using `X-Forwarded-For`. Examples: @@ -420,7 +508,13 @@ header = "X-Forwarded-For" trusted_count = 1 ``` -Note: `trusted_count` is the number of trusted proxies in front of the Signer service, but the last proxy won't add its address, so the number of skipped IPs is `trusted_count - 1`. See [MDN docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Forwarded-For#trusted_proxy_count) for more info. +:::note +`trusted_count` is the number of trusted proxies in front of the Signer service, but the last proxy won't add its address, so the number of skipped IPs is `trusted_count - 1`. See [MDN docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Forwarded-For#trusted_proxy_count) for more info. +::: + +:::warning +Only enable `[signer.reverse_proxy]` when the signer is reachable exclusively through the trusted proxy: this configuration decides whose IP gets rate-limited, so a directly reachable signer or a wrong `trusted_count` lets a client spoof the header to bypass the limit or lock other clients out. +::: ## Custom module @@ -443,15 +537,16 @@ The `cb-config.toml` file needs to be updated as follows: port = 18550 [[relays]] -url = "" +# Replace this with your relay's own URL, in the form scheme://@ +url = "https://0xa1cec75a3f0661e99299274182938151e8433c61a19222347ea1313d839229cb4ce4e3e5aa2bdeb71c8fcf1b084963c2@relay.example.com" [signer] port = 20000 -[signer.loader] +[signer.local.loader] format = "lighthouse" keys_path = "/path/to/keys" -secrets_path = "/path/to.secrets" +secrets_path = "/path/to/secrets" [metrics] enabled = true @@ -464,40 +559,39 @@ signing_id = "0x6a33a23ef26a4836979edff86c493a69b26ccf0b4a16491a815a13787657431b sleep_secs = 5 ``` -A few things to note: - -- We now added a `signer` section which will be used to create the Signer module. -- There is now a `[[modules]]` section which at a minimum needs to specify the module `id`, `type` and `docker_image`. For modules with type `commit`, which will be used to access the Signer service and request signatures for preconfs, you will also need to specify the module's unique `signing_id` (see [the propser commitment documentation](../developing/prop-commit-signing.md)). Additional parameters needed for the business logic of the module will also be here. +The `[[modules]]` section at a minimum needs to specify the module `id`, `type` and `docker_image`. For modules with type `commit`, which will be used to access the Signer service and request signatures for preconfs, you will also need to specify the module's unique `signing_id` (see [the proposer commitment documentation](../developing/prop-commit-signing.md)). You can also pass environment variables to the module with `env` (a map of variable name to value) and `env_file` (path to an environment file for the module). Additional parameters needed for the business logic of the module will also be here. -To learn more about developing modules, check out [here](/category/developing). +To learn more about developing modules, see [Commit modules](../developing/commit-modules.md). ## Vouch -[Vouch](https://github.com/attestantio/vouch) is a multi-node validator client built by [Attestant](https://www.attestant.io/). Vouch is particular in that it also integrates an MEV-Boost client to interact with relays. The Commit-Boost PBS module is compatible with the Vouch `blockrelay` since it implements the same Builder-API as relays. For example, depending on your setup and preference, you may want to fetch headers from a given relay using Commit-Boost vs using the built-in Vouch `blockrelay`. +[Vouch](https://github.com/attestantio/vouch) is a multi-node validator client built by [Attestant](https://www.attestant.io/). Vouch is particular in that it also integrates an MEV-Boost client to interact with relays. The Commit-Boost PBS service is compatible with the Vouch `blockrelay` since it implements the same Builder-API as relays. For example, depending on your setup and preference, you may want to fetch headers from a given relay using Commit-Boost vs using the built-in Vouch `blockrelay`. ### Configuration Get familiar on how to set up Vouch [here](https://github.com/attestantio/vouch/blob/master/docs/getting_started.md). -You can setup Commit-Boost with Vouch in two ways. -For simplicity, assume that in Vouch `blockrelay.listen-address: 127.0.0.0:19550` and in Commit-Boost `pbs.port = 18550`. +You can set up Commit-Boost with Vouch in two ways. +For simplicity, assume that in Vouch `blockrelay.listen-address: 127.0.0.1:19550` and in Commit-Boost `pbs.port = 18550`. #### Beacon Node to Vouch -In this setup, the BN Builder-API endpoint will be pointing to the Vouch `blockrelay` (e.g. for Lighthouse you will need the flag `--builder=http://127.0.0.0:19550`). +In this setup, the BN Builder-API endpoint will be pointing to the Vouch `blockrelay` (e.g. for Lighthouse you will need the flag `--builder=http://127.0.0.1:19550`). -Modify the `blockrelay.config` file to add Commit-Boost: +Modify the `blockrelay.config` file to add Commit-Boost to its `relays`: ```json -"relays": { - "http://127.0.0.0:18550": {} +{ + "relays": { + "http://127.0.0.1:18550": {} + } } ``` #### Beacon Node to Commit-Boost -In this setup, the BN Builder-API endpoint will be pointing to the PBS module (e.g. for Lighthouse you will need the flag `--builder=http://127.0.0.0:18550`). +In this setup, the BN Builder-API endpoint will be pointing to the PBS service (e.g. for Lighthouse you will need the flag `--builder=http://127.0.0.1:18550`). This will bypass the `blockrelay` entirely so make sure all relays are properly configured in the `[[relays]]` section. @@ -510,21 +604,51 @@ This approach could also work if you have a multi-beacon-node setup, where some - It's up to you to decide which relays will be connected via Commit-Boost (`[[relays]]` section in the `toml` config) and which via Vouch (additional entries in the `relays` field). Remember that any rate-limit will be shared across the two sidecars, if running on the same machine. - You may occasionally see a `timeout` error during registrations, especially if you're running a large number of validators in the same instance. This can resolve itself as registrations will be cleared later in the epoch when relays are less busy processing other registrations. Alternatively you can also adjust the `builderclient.timeout` option in `.vouch.yml`. -## Hot Reload +## Hot reload + +Commit-Boost can hot-reload `cb-config.toml` without restarting modules: send `POST /reload` to each module you want to reload. + +On the signer, the `/reload` and `/revoke_jwt` endpoints require admin authentication. `CB_SIGNER_ADMIN_JWT` holds the admin *secret* (an HMAC key), not a ready-to-use token: mint a short-lived HS256 admin JWT from it (claims spec: [Signer API > Admin token](../developing/prop-commit-signing.md#admin-token)). Sending the raw secret as the Bearer token fails with `401`. Commit-Boost ships no CLI helper for minting this token today; a few lines of Python do it (deps: `pip install pyjwt "eth-hash[pycryptodome]"`): + +```python +import os, time, jwt +from eth_hash.auto import keccak -Commit-Boost supports hot-reloading the configuration file. This means that you can modify the `cb-config.toml` file and apply the changes without needing to restart the modules. To do this, you need to send a `POST` request to the `/reload` endpoint on each module you want to reload the configuration. In the case the module is running in a Docker container without the port exposed (like the signer), you can use the following command: +secret = os.environ["CB_SIGNER_ADMIN_JWT"] +route = "/reload" +body = b"{}" # the request body you will send + +claims = {"admin": True, "route": route, "exp": int(time.time()) + 30} +if body: + claims["payload_hash"] = "0x" + keccak(body).hex() +print(jwt.encode(claims, secret, algorithm="HS256")) +``` + +Then send the request with the minted token. The signer's `/reload` endpoint requires a JSON body with `Content-Type: application/json` even when no overrides are sent; send an empty JSON object (`{}`) at minimum, and mint the token over the exact body you send. In the case the module is running in a Docker container without the port exposed (like the signer), you can run the request inside the container (assuming the snippet above is saved as `mint_admin_jwt.py`). In the Docker setup the admin secret lives in `.cb.env`, so export it into your shell first: ```bash -docker compose -f cb.docker-compose.yml exec cb_signer curl -X POST http://localhost:20000/reload +export $(grep CB_SIGNER_ADMIN_JWT .cb.env) +TOKEN=$(python3 mint_admin_jwt.py) +docker compose -f cb.docker-compose.yml exec cb_signer curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{}' http://localhost:20000/reload ``` -### Signer module reload +Passing the minted token on the command line is acceptable only because it expires in 30 seconds and is bound to the route and body; the admin secret itself must never appear on a command line. + +### Automatic reload (PBS only) + +In addition to the manual `/reload` endpoint, the PBS service watches the config file for changes and automatically reloads the configuration whenever the file is modified, with no restart or API call needed. If a reload fails (e.g. because of a misconfigured option), the previous configuration is kept: the watcher logs a warning and the `/reload` endpoint returns a 500 error. + +:::caution Custom PBS binaries +Custom PBS binaries only get the file watcher if they pass the real config path to `PbsState::new`; see [Extending PBS](../developing/extending-pbs.md#entry-point). `POST /reload` works either way. +::: + +### Signer service reload When the signer receives a reload request it re-reads the configuration file and environment variables, rebuilding its internal state to match: -- **New modules** added to the config are registered with the signer. -- **Removed modules** are dropped from the signer's access list. -- **JWT secrets and admin secret** are reset to their current values from the environment variables. +- New modules added to the config are registered with the signer. +- Removed modules are dropped from the signer's access list. +- JWT secrets and the admin secret are reset to their current values from the environment variables. - Any runtime changes from previous `/revoke_jwt` or `/reload` calls are reverted. The request body accepts 2 optional override parameters, applied on top of the config: @@ -532,32 +656,30 @@ The request body accepts 2 optional override parameters, applied on top of the c - `jwt_secrets`: a comma-separated list of `=` pairs to override specific module secrets. Only modules present in the config can be overridden. - `admin_secret`: a string to override the admin JWT secret. -If the body is empty, the signer state is simply synced to match the config. +With an empty JSON object body (`{}`), the signer state is simply synced to match the config. #### Common patterns **Add a new module without restarting:** +This works only if the module's JWT secret was already included in `CB_JWTS` (a comma-separated list of `=` pairs) when the signer started: the reload re-reads `CB_JWTS` from the signer's own process environment, which is fixed at startup, and fails with a 500 (`internal error`; the signer log shows `JWT secret for module X is missing`) otherwise. For a module whose secret is not yet in `CB_JWTS`, there are two Docker paths: re-run `commit-boost init` and `docker compose up -d` with the new env file so every container is recreated with consistent fresh secrets, or hand-add the module's service to the compose file and append its secret to `CB_JWTS` in `.cb.env` before restarting the signer. Re-running `init` rotates every JWT secret, including the admin secret. 1. Add the `[[modules]]` entry to `cb-config.toml`. -2. Set the new module's JWT secret in the signer's environment (`CB_SIGNER_JWT_SECRETS`). -3. Send `POST /reload` with an empty body. The signer picks up the new module from config. -4. Start the new module container with the matching JWT secret. +2. Send `POST /reload` with an empty JSON object body (`{}`). The signer picks up the new module from config. +3. Start the new module container with the matching JWT secret. **Rotate a JWT secret remotely:** -Send `POST /reload` with the new secret in the body. The module must already exist in the config. This is useful for scripted rotation without SSH access to edit config files. +Send `POST /reload` with the new secret in the body. The module must already exist in the config. Scripted rotation works without SSH access to edit config files. Unless TLS is enabled (see [TLS](#tls)), the request sends the new secret in cleartext over the network. A secret passed on a curl command line also lands in shell history and the process list; read it from a file or stdin instead. **Revoke a compromised module immediately:** Send `POST /revoke_jwt` with the module ID. This removes the module from the signer's access list without touching the config. The next `/reload` will restore the module if it is still in the config, so remove it from config as well if the revocation should be permanent. #### Footguns -- **Body overrides are not persisted.** If the signer crashes or restarts after a body-based secret rotation, it falls back to the config/environment values. The module container will still have the rotated secret and authentication will fail. To avoid this, update the environment variable to match after rotating via the body. -- **Reload reverts revocations.** If you revoke a module with `/revoke_jwt` but leave it in the config, the next `/reload` without a body override will re-add it from the config baseline. Always remove revoked modules from the config to make the revocation permanent. -- **Override validation is strict.** If the body references a module ID that does not exist in the config, the entire reload request is rejected and no changes are applied. This prevents typos from silently failing. +- Body overrides are not persisted: if the signer crashes or restarts after a body-based secret rotation, it falls back to the config/environment values. The module container will still have the rotated secret and authentication will fail. To avoid this, update the environment variable to match after rotating via the body. +- Override validation is strict: if the body references a module ID that does not exist in the config, the entire reload request is rejected and no changes are applied. This prevents typos from silently failing. ### Notes -- The hot reload feature is available for PBS modules (both default and custom) and signer module. -- Changes related to listening hosts and ports will not been applied, as it requires the server to be restarted. +- The hot reload feature is available for the PBS service (both default and custom) and the Signer service. +- Changes to listening hosts and ports are not applied, since they require a server restart. - If running in Docker containers, changes in `volumes` will not be applied, as it requires the container to be recreated. Be careful if changing a path to a local file as it may not be accessible from the container. -- Custom PBS modules may override the default behaviour of the hot reload feature to parse extra configuration fields. Check the [examples](https://github.com/Commit-Boost/commit-boost-client/blob/main/examples/status_api/src/main.rs) for more details. -- In case the reload fails (most likely because of some misconfigured option), the server will return a 500 error and the previous configuration will be kept. +- Custom PBS services may override the default behavior of the hot reload feature to parse extra configuration fields. Check the [examples](https://github.com/Commit-Boost/commit-boost-client/blob/main/examples/status_api/src/main.rs) for more details. diff --git a/docs/docs/get_started/mux-key-loaders.md b/docs/docs/get_started/mux-key-loaders.md new file mode 100644 index 000000000..6372c4cbd --- /dev/null +++ b/docs/docs/get_started/mux-key-loaders.md @@ -0,0 +1,270 @@ +--- +description: Mux (multiplexer) configuration and key loader types +--- + +# Mux key loaders + +The PBS multiplexer (or *mux*) lets you route different validators to different relay sets or timing game configurations. Instead of a single `[[relays]]` list for all your validators, you declare one or more `[[mux]]` entries that match specific validator pubkeys to custom relay and timing settings. + +A mux covers cases like a Lido or SSV node operator who sends some validators to an operator-specific relay while the rest use the global relay set, or per-group timing games: `timeout_get_header_ms` and `late_in_slot_time_ms` can be set per-mux, overriding the PBS defaults for those validators. The mux key loaders (File, URL, Registry) populate a mux's validator set from a file, an HTTP endpoint, or an on-chain registry, so you don't have to list hundreds or thousands of pubkeys by hand. + +Mux entries are an optional addition to the `[[relays]]` section. A mux affects only `get_header` requests: validator registrations and `submit_blinded_block` always go to all configured relays, global and mux alike. + +--- + +## Mux entry matching + +At startup the sidecar resolves every mux (running its loader, if any) and builds one pubkey-to-mux lookup, so entry order in the config does not matter and the mux pubkey sets must be disjoint: + +| Condition | Behavior | +|---|---| +| Pubkey belongs to exactly one mux | That mux's relays and timing config are used for `get_header`; registrations and `submit_blinded_block` still go to all relays | +| Pubkey appears in more than one mux | Startup error: `duplicate validator pubkey in muxes` | +| Pubkey doesn't belong to any mux | Falls through to global `[[relays]]` | +| A mux has no pubkeys (empty set) | Startup error: each mux must have at least one pubkey | +| A mux has no relays | Startup error: each mux must have at least one relay | + +```toml +# Global relays, used for validators not matching any mux +[[relays]] +id = "global-relay" +url = "..." + +# A mux entry; its pubkeys must not appear in any other mux +[[mux]] +id = "timing-sensitive" +validator_pubkeys = [ + "0x80c7f782b2467c5898c5516a8b6595d75623960b4afc4f71ee07d40985d20e117ba35e7cd352a3e75fb85a8668a3b745", +] + +# A relay used by this mux +[[mux.relays]] +id = "fast-relay" +url = "..." + +# Another relay used by this mux +[[mux.relays]] +id = "robust-relay" +url = "..." + +# ... +# Multiple muxes can be defined repeating this pattern +``` + +The pubkey set for a mux can come from two sources combined: +1. **Inline `validator_pubkeys`**: a list of hex-prefixed BLS pubkeys in the config file itself. +2. **A loader plugin**: loads additional keys from a file, URL, or on-chain registry. Keys from the loader are merged into the mux's pubkey set *before* the disjointness check runs, so a key pulled in by a loader can collide with one you listed inline in another mux. + +--- + +## Key loaders + +Key loaders are how you populate a mux with validator pubkeys without listing them manually. They are configured via the `loader` field inside a `[[mux]]` entry. + +### File loader + +Loads pubkeys from a flat JSON file on disk. + +The file is a JSON array of hex-prefixed BLS public key strings. + +```json +[ + "0x8160998addda06f2956e5d1945461f33dbc140486e972b96f341ebf2bdb553a0e3feb127451f5332dd9e33469d37ca67", + "0x87b5dc7f78b68a7b5e7f2e8b9c2115f968332cbf6fc2caaaaa2c9dc219a58206b72c924805f2278c58b55790a2c3bf17", + "0x89e2f50fe5cd07ed2ff0a01340b2f717aa65cced6d89a79fdecc1e924be5f4bbe75c11598bb9a53d307bb39b8223bc52" +] +``` + +Relative paths resolve against the sidecar's working directory, not the config file's location; absolute paths are recommended for binary deployments. + +```toml +[[mux]] +id = "my-mux" +loader = "./path/to/keys.json" + +[[mux.relays]] +id = "my-relay" +url = "..." +``` + +The path can be overridden at runtime via `CB_MUX_PATH_{id}` where `{id}` is the mux identifier, verbatim. For a mux with `id = "lido-mux"`, the variable is `CB_MUX_PATH_lido-mux`. + +```bash +env CB_MUX_PATH_lido-mux="/path/to/override.json" commit-boost pbs +``` + +Hyphenated mux ids cannot be set with bash `export` (a hyphen is not valid in a shell identifier); use `env` as above, a compose `environment:` entry, or an underscore-only mux id. + +--- + +### URL loader + +Loads pubkeys from an HTTP(S) endpoint returning the same JSON array format as the File loader. + +```toml +[[mux]] +id = "url-mux" +loader = { url = "https://keys.example.com/validators.json" } + +[[mux.relays]] +id = "my-relay" +url = "..." +``` + +HTTPS is recommended; plain HTTP works but triggers a warning at startup. + +The loader makes a one-shot GET request with no retry logic. The timeout is controlled by `http_timeout_seconds` in the `[pbs]` section (default: 10s). The response body is read (up to a 10 MiB limit) and parsed as JSON; larger responses fail the load. + +--- + +### Registry loader + +Loads validator pubkeys from an on-chain or network registry. + +Three registries are currently supported: + +| Registry | `registry` value | Key source | Authentication | +|---|---|---|---| +| Lido | `"lido"` | On-chain contract via RPC | RPC URL (from `[pbs]` config) | +| SSV | `"ssv"` | SSV node API or public API | SSV API URLs (from `[pbs]` config) | +| Stader | `"stader"` | On-chain contract via RPC | RPC URL (from `[pbs]` config) | + +Registry entries must be unique within their registry type (one Lido entry per node operator ID, one SSV entry per node operator ID, one Stader entry per pool and node operator ID); a Lido and an SSV entry may share a node operator ID, since the registries are independent. + +#### Lido registry + +Reads validator pubkeys from Lido's on-chain `NodeOperatorsRegistry` or `CSModule registry`, depending on the module type. The sidecar connects to the configured RPC endpoint and calls the contract's `getSigningKeys` method with pagination. + +`rpc_url` must be set in the `[pbs]` configuration. + +```toml +[pbs] +port = 18550 +rpc_url = "https://ethereum-rpc.publicnode.com" + +[[mux]] +id = "lido-mux" +loader = { registry = "lido", node_operator_id = 8, lido_module_id = 1 } + +[[mux.relays]] +id = "lido-relay" +url = "..." +``` + +**Fields:** + +| Field | Type | Required | Description | +|---|---|---|---| +| `registry` | string | Yes | Must be `"lido"` | +| `node_operator_id` | integer | Yes | Lido node operator ID | +| `lido_module_id` | integer | No (default: `1`) | Lido staking module ID | +| `enable_refreshing` | boolean | No (default: `false`) | Whether to periodically refresh keys at runtime (see below) | + +**Chain support:** + +| Chain | `lido_module_id` | Module type | Contract type | +|---|---|---|---| +| Mainnet | 1 | Curated (NodeOperatorsRegistry) | `NodeOperatorsRegistry` | +| Mainnet | 2 | SimpleDVT | `NodeOperatorsRegistry` | +| Mainnet | 3 | Community Staking (CSM) | `CSModule` | +| Holesky | 1 | Curated (NodeOperatorsRegistry) | `NodeOperatorsRegistry` | +| Holesky | 2 | SimpleDVT | `NodeOperatorsRegistry` | +| Holesky | 3 | Sandbox | `NodeOperatorsRegistry` | +| Holesky | 4 | Community Staking (CSM) | `CSModule` | +| Hoodi | 1 | Curated (NodeOperatorsRegistry) | `NodeOperatorsRegistry` | +| Hoodi | 2 | SimpleDVT | `NodeOperatorsRegistry` | +| Hoodi | 3 | Sandbox | `NodeOperatorsRegistry` | +| Hoodi | 4 | Community Staking (CSM) | `CSModule` | +| Sepolia | 1 | | `NodeOperatorsRegistry` | + +The sidecar picks the right contract automatically based on chain and module id. + +When `enable_refreshing = true`, the sidecar periodically re-fetches keys from the on-chain registry at runtime. New validators that register with the node operator are picked up automatically without a restart, and validators removed from the registry are dropped from the mux (falling back to the global relays) on the same refresh cycle. The refresh interval is controlled by `mux_registry_refresh_interval_seconds` in the `[pbs]` configuration (default: `384` seconds, i.e. one epoch; must be greater than 0), and applies to all registry muxes with refreshing enabled. + +--- + +#### SSV registry + +Loads validator pubkeys from the SSV network. The sidecar first tries to fetch keys from your local SSV node API. If that fails, it falls back to the public SSV API. + +No `[pbs]` settings are required: `ssv_node_api_url` and `ssv_public_api_url` are optional and default to `http://localhost:16000/v1/` (node API) and `https://api.ssv.network/api/v4/` (public API). Set them only to point at a different node or API server. + +```toml +[pbs] +port = 18550 +ssv_node_api_url = "http://localhost:16000/v1/" +ssv_public_api_url = "https://api.ssv.network/api/v4/" + +[[mux]] +id = "ssv-mux" +loader = { registry = "ssv", node_operator_id = 200 } + +[[mux.relays]] +id = "ssv-relay" +url = "..." +``` + +**Fields:** + +| Field | Type | Required | Description | +|---|---|---|---| +| `registry` | string | Yes | Must be `"ssv"` | +| `node_operator_id` | integer | Yes | SSV node operator ID | +| `enable_refreshing` | boolean | No (default: `false`) | Whether to periodically refresh keys at runtime | + +The loader tries two API sources in order: + +1. **SSV node API** (preferred): `GET {ssv_node_api_url}validators` with a JSON body `{"operators": [node_operator_id]}`. Response contains a `data` array of validators with hex-encoded `public_key` fields. +2. **Public API** (fallback): `GET {ssv_public_api_url}{chain}/validators/in_operator/{node_operator_id}?perPage=100&page={page}` with pagination. Response contains a `validators` array and `pagination` object. + +If the node API call fails (timeout, connection error, etc.), the sidecar logs a warning and falls back to the public API. + +The public-API fallback supports Mainnet, Holesky, and Hoodi only; the node API path is chain-agnostic, so on other chains the loader works only while your local SSV node API is reachable. + +--- + +#### Stader registry + +Reads validator pubkeys from Stader's on-chain node registry. The sidecar connects to the configured RPC endpoint and queries the registry contract for the pool you specify. + +`rpc_url` must be set in the `[pbs]` configuration, and `stader_pool` must be set in the mux config. + +```toml +[pbs] +port = 18550 +rpc_url = "https://ethereum-rpc.publicnode.com" + +[[mux]] +id = "stader-mux" +loader = { registry = "stader", node_operator_id = 200, stader_pool = "permissioned" } + +[[mux.relays]] +id = "stader-relay" +url = "..." +``` + +**Fields:** + +| Field | Type | Required | Description | +|---|---|---|---| +| `registry` | string | Yes | Must be `"stader"` | +| `node_operator_id` | integer | Yes | Stader node operator ID | +| `stader_pool` | string | Yes | Stader staking pool: `"permissioned"` or `"permissionless"` | +| `enable_refreshing` | boolean | No (default: `false`) | Whether to periodically refresh keys at runtime | + +Stader is supported on Mainnet only. + +--- + +## Reference config + +For a complete working example with multiple mux entries (File loader, Lido registry, SSV registry, and Stader registry), see: + +- [`examples/configs/pbs_mux.toml`](https://github.com/Commit-Boost/commit-boost-client/blob/main/examples/configs/pbs_mux.toml) + +--- + +## See also + +- [Configuration reference](./configuration.md): full config field listing +- [Signer API](../developing/prop-commit-signing.md#api-quickstart): signing API quickstart and authentication diff --git a/docs/docs/get_started/overview.md b/docs/docs/get_started/overview.md index b57195674..b9fafd616 100644 --- a/docs/docs/get_started/overview.md +++ b/docs/docs/get_started/overview.md @@ -2,19 +2,19 @@ description: Initial setup --- -# Overview +# Getting started -Commit-Boost is primarily based on [Docker](https://www.docker.com/) to enable modularity, sandboxing and cross-platform compatibility. It is also possible to run Commit-Boost [natively](/get_started/running/binary) without Docker. +Commit-Boost is primarily based on [Docker](https://www.docker.com/) to enable modularity, sandboxing and cross-platform compatibility. It is also possible to run Commit-Boost [natively](./running/binary.md) without Docker. -Each component roughly maps to a container: from a single `.toml` config file, the node operator can specify which modules they want to run, and Commit-Boost takes care of spinning up the services and creating links between them. +Each component roughly maps to a container: from a single `.toml` config file, the node operator can specify which services they want to run, and Commit-Boost takes care of spinning up the services and creating links between them. Commit-Boost ships with two core services: -- A PBS module which implements the [BuilderAPI](https://ethereum.github.io/builder-specs/) for [MEV Boost](https://docs.flashbots.net/flashbots-mev-boost/architecture-overview/specifications). -- A signer module, which implements the [Signer API](/api) and provides the interface for modules to request proposer commitments. +- A PBS service which implements the [BuilderAPI](https://ethereum.github.io/builder-specs/) for [MEV Boost](https://docs.flashbots.net/flashbots-mev-boost/architecture-overview/specifications). +- A Signer service, which implements the [Signer API](/api) and provides the interface for modules to request proposer commitments. ## Setup -The Commit-Boost program can create a dynamic `docker-compose` file, with services and ports already set up. +The Commit-Boost binary can create a dynamic `docker-compose` file, with services and ports already set up. Whether you're using Docker or running the binaries natively, you can compile from source directly from the repo, or download binaries and fetch docker images from the official releases. @@ -24,49 +24,6 @@ Find the latest releases at https://github.com/Commit-Boost/commit-boost-client/ The services are also published at [each release](https://github.com/orgs/Commit-Boost/packages?repo_name=commit-boost-client). -### From source +## Build from source -Requirements: - -- Rust 1.91 - -:::note -Run `rustup update` to update Rust and Cargo to the latest version -::: - -```bash -# Pull the repo -git clone https://github.com/Commit-Boost/commit-boost-client - -# Stable branch has the latest released version -git checkout stable - -# Init submodules -git submodule update --init --recursive -``` - -:::note -If you get an `openssl` related error try running: `apt-get update && apt-get install -y openssl ca-certificates libssl3 libssl-dev build-essential pkg-config` -::: - -Now, build the binary, which will be stored in `build//`, for example `build/206658b/linux_amd64/`: - -```bash -just build-bin $(git rev-parse --short HEAD) -``` - -You can confirm the binary was built successfully by navigating to the build directory and checking its version: -```bash -./commit-boost --version -``` - -### Docker - -Building the service images requires the binary to be built using the above instructions first, since it will be copied into those images. Once it's built, create the images with the following: - -```bash -just build-pbs-img $(git rev-parse --short HEAD) -just build-signer-img $(git rev-parse --short HEAD) -``` - -This will create two local images called `commit_boost/pbs:` and `commit_boost/signer:` for the PBS and Signer services respectively. Make sure to use these images in the `docker_image` field in the `[pbs]` and `[signer]` sections of the `.toml` config file, respectively. +To build the binary and Docker images yourself, see [Building from source](./building.md). diff --git a/docs/docs/get_started/running/binary.md b/docs/docs/get_started/running/binary.md index 8f51fe657..006e14b0b 100644 --- a/docs/docs/get_started/running/binary.md +++ b/docs/docs/get_started/running/binary.md @@ -5,42 +5,45 @@ description: Run Commit-Boost modules natively # Binary :::warning -Running the modules natively means you opt out of the security guarantees made by Docker and it's up to you how to setup and ensure the modules run safely. +Running the modules natively means you opt out of the security guarantees made by Docker, and it is up to you to set up the modules and ensure they run safely. ::: ## Setup Get the binary of the module either by compiling from source or by downloading a [published release](https://github.com/Commit-Boost/commit-boost-client/releases). -Modules need some environment variables to work correctly. +Services need environment variables to work correctly. ### Common - `CB_CONFIG`: required, path to the `.toml` config file. -- `CHAIN_SPEC_ENV`: optional, path to a chain spec file. This will override the `[chain]` field in the `.toml` config. +- `CB_CHAIN_SPEC`: optional, path to a chain spec file. It overrides the `path` of the top-level `chain` key in the `.toml` config, and **only** when `chain` uses the spec-file form `chain = { genesis_time_secs = ..., path = "..." }` (see [Custom chains](../configuration.md#custom-chains)). If `chain` is a network name (e.g. `chain = "Holesky"`) or the fully-inline custom object, this variable is silently ignored. - `CB_METRICS_PORT`: optional, port where to expose the `/metrics` endpoint for Prometheus. - `CB_LOGS_DIR`: optional, directory to store logs. This will override the directory in the `.toml` config. -### PBS Module +### PBS Service -- `CB_PBS_ENDPOINT`: optional, override to specify the `IP:port` endpoint where the PBS module will open the port for the beacon node. -- `CB_MUX_PATH_{ID}`: optional, override where to load mux validator keys for mux with `id=\{ID\}`. +- `CB_PBS_ENDPOINT`: optional, override to specify the `IP:port` endpoint where the PBS service will open the port for the beacon node. +- `CB_MUX_PATH_{ID}`: optional, override where to load mux validator keys for mux with `id={ID}`. -### Signer Module +### Signer Service -- `CB_SIGNER_ADMIN_JWT`: secret to use for admin JWT. +- `CB_JWTS`: required (the Signer service will not start without it), comma-separated list of `module_id=jwt_secret` pairs for module authentication. +- `CB_SIGNER_ADMIN_JWT`: required, secret to use for admin JWT. +- `CB_SIGNER_JWT_AUTH_FAIL_LIMIT`: optional, override the number of failed JWT auth attempts before rate-limiting a client (default: `3`). +- `CB_SIGNER_JWT_AUTH_FAIL_TIMEOUT_SECONDS`: optional, override the rate-limit timeout window in seconds (default: `300`). - `CB_SIGNER_ENDPOINT`: optional, override to specify the `IP:port` endpoint to bind the signer server to. -- `CB_SIGNER_TLS_CERTIFICATES`: path to the TLS certificates for the server. +- `CB_SIGNER_TLS_CERTIFICATES`: optional, override of the TLS certificates directory (must contain `cert.pem` and `key.pem`). Only used when the signer's `tls_mode` is set to a certificate path. - For loading keys we currently support: - `CB_SIGNER_LOADER_FILE`: path to a `.json` with plaintext keys (for testing purposes only). - - `CB_SIGNER_LOADER_FORMAT`, `CB_SIGNER_LOADER_KEYS_DIR` and `CB_SIGNER_LOADER_SECRETS_DIR`: paths to the `keys` and `secrets` directories or files (ERC-2335 style keystores, see [Signer config](../configuration/#signer-module) for more info). + - `CB_SIGNER_LOADER_KEYS_DIR` and `CB_SIGNER_LOADER_SECRETS_DIR`: paths to the `keys` and `secrets` directories or files (ERC-2335 style keystores, see [Signer config](../configuration.md#signer-service) for more info). - For storing proxy keys we currently support: - `CB_PROXY_STORE_DIR`: directory where proxy keys and delegations will be saved in plaintext (for testing purposes only). - - `CB_PROXY_KEYS_DIR` and `CB_PROXY_SECRETS_DIR`: paths to the `keys` and `secrets` directories or files (ERC-2335 style keystores, see [Proxy keys store](../configuration/#proxy-keys-store) for more info). -- For Dirk remote signer the following envs are available (see [Dirk config](../configuration/#dirk) for more info): - - `CB_SIGNER_DIRK_CERT_FILE`: required, path to the client certificate file. - - `CB_SIGNER_DIRK_KEY_FILE`: required, path to the client key file. - - `CB_SIGNER_DIRK_SECRETS_DIR`: required, path to the secrets directory. + - `CB_PROXY_KEYS_DIR` and `CB_PROXY_SECRETS_DIR`: paths to the `keys` and `secrets` directories or files (ERC-2335 style keystores, see [Proxy keys store](../configuration.md#proxy-keys-store) for more info). +- For Dirk remote signer the following envs are available (see [Dirk config](../configuration.md#dirk) for more info): + - `CB_SIGNER_DIRK_CERT_FILE`: optional, override of the `cert_path` in the `[signer.dirk]` config, path to the client certificate file. + - `CB_SIGNER_DIRK_KEY_FILE`: optional, override of the `key_path` in the `[signer.dirk]` config, path to the client key file. + - `CB_SIGNER_DIRK_SECRETS_DIR`: optional, override of the `secrets_path` in the `[signer.dirk]` config, path to the secrets directory. - `CB_SIGNER_DIRK_CA_CERT_FILE`: optional, path to the CA certificate file. ### Modules @@ -49,19 +52,23 @@ Modules need some environment variables to work correctly. #### Commit modules -- `CB_SIGNER_URL`: required, url to the signer module server. -- `CB_SIGNER_JWT`: required, jwt to use for signature requests. +- `CB_SIGNER_URL`: required, url to the Signer service server. +- `CB_SIGNER_JWT`: required, the module's pre-shared JWT secret, from which the module mints per-request tokens. Must be identical to this module's entry in the signer's `CB_JWTS`; generate one per module (e.g. `openssl rand -hex 32`). See [Module JWT](../../developing/prop-commit-signing.md#module-jwt). Modules might also have additional envs required, which should be detailed by the maintainers. ## Start -After creating the `cb-config.toml` file, setup the required envs and run the binary. For example: +After creating the `cb-config.toml` file, set up the required envs and run the binary. For example: ```bash CB_CONFIG=./cb-config.toml commit-boost pbs ``` -## Security +Or for the Signer service: -Running the modules natively means you opt out of the security guarantees made by Docker and it's up to you how to setup and ensure the modules run safely. +```bash +CB_CONFIG=./cb-config.toml CB_JWTS="MY_MODULE=" CB_SIGNER_ADMIN_JWT="" commit-boost signer +``` + +For a worked signer startup, see [Verifying the Signer Service](../building.md#verifying-the-signer-service). diff --git a/docs/docs/get_started/running/docker.md b/docs/docs/get_started/running/docker.md index 81fd9f850..23139656b 100644 --- a/docs/docs/get_started/running/docker.md +++ b/docs/docs/get_started/running/docker.md @@ -3,7 +3,7 @@ description: Run Commit-Boost with Docker --- # Docker -The Commit-Boost program generates a dynamic `docker-compose.yml` file using the provided `.toml` config file. This is the recommended approach as Docker provides sandboxing of the containers from the rest of your system. +The Commit-Boost program generates a `cb.docker-compose.yml` file from the provided `.toml` config file. This is the recommended approach as Docker provides sandboxing of the containers from the rest of your system. ## Init @@ -11,43 +11,48 @@ First run: ```bash commit-boost init --config cb-config.toml ``` -This will create up to three files: +The optional `-o`/`--output` flag sets where the files are written (default: the current directory). + +This will create two files: - `cb.docker-compose.yml` which contains the full setup of the Commit-Boost services. -- `.cb.env` with local env variables, including JWTs for modules, only created if the signer module is enabled. -- `target.json` which enables dynamic discovery of services for metrics scraping via Prometheus, only created if metrics are enabled. +- `.cb.env` with local env variables, including JWTs for modules, only created if the Signer service is enabled. ## Start To start Commit-Boost run: ```bash -docker compose --env-file ".cb.env" -f ".cb.docker-compose.yml" up -d +docker compose --env-file ".cb.env" -f "cb.docker-compose.yml" up -d ``` -This will run all the configured services, including PBS, signer and modules (if any). +:::note +If only the PBS service is configured, no `.cb.env` file is generated and the `--env-file ".cb.env"` flag must be omitted (Docker Compose errors if the file doesn't exist). In general, use the exact command that `commit-boost init` prints. The same applies to the `logs` and `down` commands below. +::: + +This will run all the configured services, including PBS, signer and commit modules (if any). The MEV-Boost server will be exposed at `pbs.port` from the config, `18550` in our example. You'll need to point your CL/Validator client to this port to be able to source blocks from the builder market. ## Logs To check the logs, run: ```bash -docker compose --env-file ".cb.env" -f ".cb.docker-compose.yml" logs -f +docker compose --env-file ".cb.env" -f "cb.docker-compose.yml" logs -f ``` -This will currently show all logs from the different services via the Docker logs interface. Logs are also optionally saved to file, depending on your `[logs]` configuration. +This will currently show all logs from the different services via the Docker logs interface. Logs are also optionally saved to file, depending on your [`[logs]` configuration](../configuration.md#logs). ## Stop To stop all the services and cleanup, simply run: ```bash -docker compose --env-file ".cb.env" -f ".cb.docker-compose.yml" down +docker compose --env-file ".cb.env" -f "cb.docker-compose.yml" down ``` This will wind down all services and clear internal networks and file mounts. ## Example with PBS Only -This section provides an example of a configuration where only the PBS service is run with its default configuration, and the Docker compose file produced by that configuration. +A minimal configuration that runs only the PBS service with its defaults, and the Docker compose file `init` produces from it. -All of PBS's parameters are controlled via the [Commit-Boost TOML configuration file](../configuration.md); the service cannot currently be controlled with command-line arguments. Therefore it is crucial to ensure that you have a configuration file present with all of the settings you require *before* starting the service, as this file will be mounted within the Docker container as a volume in read-only mode. +All of PBS's parameters are controlled via the [Commit-Boost TOML configuration file](../configuration.md); the service cannot currently be controlled with command-line arguments. Make sure the configuration file exists with all of the settings you require *before* starting the service, as it is mounted into the Docker container as a read-only volume. Below is a simple configuration for running only the PBS service on the Hoodi network with two relays: @@ -55,40 +60,38 @@ Below is a simple configuration for running only the PBS service on the Hoodi ne chain = "Hoodi" [pbs] -docker_image = "ghcr.io/commit-boost/commit-boost:v0.8.0" +docker_image = "ghcr.io/commit-boost/commit-boost:v0.10.0" relay_check = true wait_all_registrations = true [[relays]] id = "abc" -url = "http://0xa1cec75a3f0661e99299274182938151e8433c61a19222347ea1313d839229cb4ce4e3e5aa2bdeb71c8fcf1b084963c2@abc.xyz" +url = "https://0xa1cec75a3f0661e99299274182938151e8433c61a19222347ea1313d839229cb4ce4e3e5aa2bdeb71c8fcf1b084963c2@abc.example.com" [[relays]] id = "def" -url = "http://0xa1cec75a3f0661e99299274182938151e8433c61a19222347ea1313d839229cb4ce4e3e5aa2bdeb71c8fcf1b084963c2@def.xyz" +url = "https://0xa1cec75a3f0661e99299274182938151e8433c61a19222347ea1313d839229cb4ce4e3e5aa2bdeb71c8fcf1b084963c2@def.example.com" ``` -Note that there are many more parameters that Commit-Boost supports, but they are all omitted and thus will use their default options. For a full description of the default options within the config file, go to the [annotated configuration example](../../../../config.example.toml). +There are many more parameters that Commit-Boost supports, but they are all omitted here and thus will use their default options. For a full description of the default options within the config file, go to the [annotated configuration example](https://github.com/Commit-Boost/commit-boost-client/blob/main/config.example.toml). -The relays here are placeholder for the sake of the example; for a list of actual relays, visit [the EthStaker relay list](https://github.com/eth-educators/ethstaker-guides/blob/main/MEV-relay-list.md). +The relays here are placeholders for the sake of the example; for a list of actual relays, visit [the EthStaker relay list](https://github.com/ethstaker/ethstaker-guides/blob/main/MEV-relay-list.md). ### Commit-Boost Init Output -Run `commit-boost init --config cb-config.toml` with the above configuration, the program will produce the following Docker Compose file: +Run `commit-boost init --config cb-config.toml` with the above configuration. The program will produce the following Docker Compose file: ``` services: cb_pbs: - command: - - pbs healthcheck: test: curl -f http://localhost:18550/eth/v1/builder/status interval: 30s timeout: 5s retries: 3 start_period: 5s - image: ghcr.io/commit-boost/commit-boost:v0.8.0 + image: ghcr.io/commit-boost/commit-boost:v0.10.0 container_name: cb_pbs ports: - 127.0.0.1:18550:18550 @@ -97,6 +100,8 @@ services: CB_PBS_ENDPOINT: 0.0.0.0:18550 volumes: - ./cb-config.toml:/cb-config.toml:ro + command: + - pbs ``` This will run the PBS service in a container named `cb_pbs`. @@ -106,7 +111,9 @@ This will run the PBS service in a container named `cb_pbs`. The program creates a read-only volume binding for the config file, which the PBS service needs to run. The Docker compose file that it creates with the `init` command, `cb.docker-compose.yml`, will be placed into your current working directory when you run the program. The volume source will be specified as a *relative path* to that working directory, so it's ideal if the config file is directly within your working directory (or a subdirectory). If you need to specify an absolute path for the config file, you can adjust the `volumes` entry within the Docker compose file manually after its creation. -Since this is a volume, the PBS service container will reload the file from disk any time it's restarted. That means you can change the file any time after the Docker compose file is created to tweak PBS's parameters, but it also means the config file must stay in the same location; if you move it, the PBS container won't be able to mount it anymore and fail to start unless you manually adjust the volume's source location. +Since this is a volume, the PBS service sees changes to the file: the stock PBS image watches the config file and [automatically reloads the configuration](../configuration.md#automatic-reload-pbs-only) whenever the file is modified, without needing a restart. That means you can change the file any time after the Docker compose file is created to tweak PBS's parameters, but it also means the config file must stay in the same location; if you move it, the PBS container won't be able to mount it anymore and will fail to start unless you manually adjust the volume's source location. + +Custom PBS images may not auto-reload; see [Extending PBS](../../developing/extending-pbs.md#entry-point). ### Networking @@ -120,8 +127,8 @@ host = "0.0.0.0" to the `[pbs]` section in the configuration. This will cause the resulting `ports` entry in the Docker compose file to become: ``` - ports: - - 0.0.0.0:18550:18550 +ports: + - 0.0.0.0:18550:18550 ``` though you will need to add an entry to your local machine's firewall software (if applicable) for other machines to access it. @@ -129,15 +136,15 @@ though you will need to add an entry to your local machine's firewall software ( Currently, the program will always export the PBS service's API port in one of these two ways. If you don't want to expose it at all, so it can only be accessed by other Docker containers running within Docker's internal network, you will need to manually remove the `ports` entry from the Docker compose file after it's been created: ``` - ports: [] +ports: [] ``` -## Example with PBS, Signer, and a Signer Module +## Example with PBS, Signer, and a Commit Module -In this scenario we will be running the PBS service, the Signer service, and a module (`DA_COMMIT`) that interacts with the Signer service's API. +In this scenario we will be running the PBS service, the Signer service, and a commit module (`DA_COMMIT`) that interacts with the Signer service's API. -All of both PBS's and the Signer's parameters are controlled via the [Commit-Boost TOML configuration file](../configuration.md); the services cannot currently be controlled with command-line arguments. Therefore it is crucial to ensure that you have a configuration file present with all of the settings you require *before* starting the services, as this file will be mounted within the Docker containers as a volume in read-only mode. +The same configuration-file rules as the [PBS-only example](#example-with-pbs-only) apply. Below is a simple configuration for running only the three modules on the Hoodi network with two relays, extended from the prior scenario above: @@ -145,19 +152,20 @@ Below is a simple configuration for running only the three modules on the Hoodi chain = "Hoodi" [pbs] -docker_image = "ghcr.io/commit-boost/commit-boost:v0.8.0" +docker_image = "ghcr.io/commit-boost/commit-boost:v0.10.0" relay_check = true wait_all_registrations = true [[relays]] id = "abc" -url = "http://0xa1cec75a3f0661e99299274182938151e8433c61a19222347ea1313d839229cb4ce4e3e5aa2bdeb71c8fcf1b084963c2@abc.xyz" +url = "https://0xa1cec75a3f0661e99299274182938151e8433c61a19222347ea1313d839229cb4ce4e3e5aa2bdeb71c8fcf1b084963c2@abc.example.com" [[relays]] id = "def" -url = "http://0xa1cec75a3f0661e99299274182938151e8433c61a19222347ea1313d839229cb4ce4e3e5aa2bdeb71c8fcf1b084963c2@def.xyz" +url = "https://0xa1cec75a3f0661e99299274182938151e8433c61a19222347ea1313d839229cb4ce4e3e5aa2bdeb71c8fcf1b084963c2@def.example.com" [signer] +docker_image = "ghcr.io/commit-boost/commit-boost:v0.10.0" port = 20000 [signer.local.loader] @@ -169,19 +177,18 @@ secrets_path = "./secrets" id = "DA_COMMIT" type = "commit" docker_image = "test_da_commit" +signing_id = "0x6a33a23ef26a4836979edff86c493a69b26ccf0b4a16491a815a13787657431b" sleep_secs = 5 ``` -Note that there are many more parameters that Commit-Boost supports, but they are all omitted and thus will use their default options. For a full description of the default options within the config file, go to the [annotated configuration example](../../../../config.example.toml). - -The relays here are placeholder for the sake of the example; for a list of actual relays, visit [the EthStaker relay list](https://github.com/eth-educators/ethstaker-guides/blob/main/MEV-relay-list.md). - In this scenario there are two folders in the same directory as the configuration file (the working directory): `keys` and `secrets`. These correspond to the folders containing the [EIP-2335 keystores](../configuration.md#local-signer) and secrets in Lighthouse format. For your own keys, adjust the `format` parameter within the configuration and directory paths accordingly. +If either the `docker_image` under the `[signer]` or `[pbs]` is left unspecified, it defaults to `ghcr.io/commit-boost/commit-boost:latest`. Make sure to specify both if you intend to use versions other than the latest release. + ### Commit-Boost Init Output -Run `commit-boost init --config cb-config.toml` with the above configuration, the program will produce two files: +Run `commit-boost init --config cb-config.toml` with the above configuration. The program will produce two files: - `cb.docker-compose.yml` - `.cb.env` @@ -206,15 +213,13 @@ services: cb_signer: condition: service_healthy cb_pbs: - command: - - pbs healthcheck: test: curl -f http://localhost:18550/eth/v1/builder/status interval: 30s timeout: 5s retries: 3 start_period: 5s - image: ghcr.io/commit-boost/commit-boost:latest + image: ghcr.io/commit-boost/commit-boost:v0.10.0 container_name: cb_pbs ports: - 127.0.0.1:18550:18550 @@ -223,22 +228,24 @@ services: CB_PBS_ENDPOINT: 0.0.0.0:18550 volumes: - ./cb-config.toml:/cb-config.toml:ro - cb_signer: command: - - signer + - pbs + cb_signer: healthcheck: - test: curl -f http://localhost:20000/status + test: curl -k -f http://cb_signer:20000/status interval: 30s timeout: 5s retries: 3 start_period: 5s - image: ghcr.io/commit-boost/commit-boost:latest + image: ghcr.io/commit-boost/commit-boost:v0.10.0 container_name: cb_signer ports: - 127.0.0.1:20000:20000 environment: CB_CONFIG: /cb-config.toml CB_JWTS: ${CB_JWTS} + CB_SIGNER_ADMIN_JWT: ${CB_SIGNER_ADMIN_JWT} + CB_SIGNER_TLS_CERTIFICATES: /certs CB_SIGNER_ENDPOINT: 0.0.0.0:20000 CB_SIGNER_LOADER_KEYS_DIR: /keys CB_SIGNER_LOADER_SECRETS_DIR: /secrets @@ -248,10 +255,11 @@ services: - ./secrets:/secrets:ro networks: - signer_network + command: + - signer networks: signer_network: driver: bridge - ``` This will create three Docker containers when executed: @@ -263,11 +271,14 @@ This will create three Docker containers when executed: Finally, the `.cb.env` file produced will look like this: ``` -CB_JWT_DA_COMMIT=mwDSSr7chwy9eFf7RhedBoyBtrwFUjSQ -CB_JWTS=DA_COMMIT=mwDSSr7chwy9eFf7RhedBoyBtrwFUjSQ +CB_JWT_DA_COMMIT=hJ0bV40p... +CB_JWTS=DA_COMMIT=hJ0bV40p... +CB_SIGNER_ADMIN_JWT=WbdxlH32... ``` -The Signer service needs JWT authentication from each of its modules. The program creates these and embeds them into the containers via environment variables automatically for convenience. This is demonstrated for the Signer module within the `environment` compose block: the `CB_JWTS: ${CB_JWTS}` forwards the `CB_JWTS` environment variable that's present when running Docker compose. The program requests that you do so via the command `docker compose --env-file "./.cb.env" -f "./cb.docker-compose.yml" up -d`; the `--env-file "./.cb.env"` handles loading the program's JWT output into this environment variable. +The values shown are truncated examples; `init` generates fresh random secrets on every run. The file contains live secrets: restrict it with `chmod 600 .cb.env` and keep it out of backups and version control. + +The Signer service needs JWT authentication from each of its modules. The program creates these and embeds them into the containers via environment variables automatically for convenience. This is demonstrated for the Signer service within the `environment` compose block: the `CB_JWTS: ${CB_JWTS}` forwards the `CB_JWTS` environment variable that's present when running Docker compose. The program requests that you do so via the command `docker compose --env-file "./.cb.env" -f "./cb.docker-compose.yml" up -d`; the `--env-file "./.cb.env"` handles loading the program's JWT output into this environment variable. Similarly, for the `cb_da_commit` module, the `CB_SIGNER_JWT: ${CB_JWT_DA_COMMIT}` line within its `environment` block will set the JWT that it should use to authenticate with the Signer service. @@ -279,31 +290,6 @@ As with the PBS-only example, the configuration file is placed into a read-only ### Networking -The program will force both the PBS and Signer API endpoints to bind to `0.0.0.0` within Docker's internal network so other Docker containers can access them, but it will only expose the API port (default `18550` for PBS and `20000` for the Signer) to `127.0.0.1` on your host machine. That way any processes running on the same machine can access them on their respective ports. If you want to open the ports for access across your entire network, not just your local machine, you can add the line: - -``` -host = "0.0.0.0" -``` - -to both the `[pbs]` and `[signer]` sections in the configuration. This will cause the resulting `ports` entries in the Docker compose file to become: - -``` - cb_pbs: - ... - ports: - - 0.0.0.0:18550:18550 +The same networking rules as the [PBS-only example](#networking) apply to both services; the Signer's API port (default `20000`) is exposed to `127.0.0.1` alongside PBS's `18550`. To open them to your network, add `host = "0.0.0.0"` to both the `[pbs]` and `[signer]` sections, or remove the `ports` entries from the compose file to keep them Docker-internal. - - cb_signer: - ... - ports: - - 0.0.0.0:20000:20000 -``` - -though you will need to add entries to your local machine's firewall software (if applicable) for other machines to access them. - -Currently, the program will always export the PBS and Signer services' API ports in one of these two ways. If you don't want to expose them at all, so they can only be accessed by other Docker containers running within Docker's internal network, you will need to manually remove the `ports` entries from the Docker compose files after they've been created: - -``` - ports: [] -``` +Unlike the [metrics ports](../configuration.md#metrics), both of these ports carry authenticated APIs, and the signer fronts your validator keys over plain HTTP by default, so open them beyond localhost only behind a firewall and, for the signer, with [TLS enabled](../configuration.md#tls). diff --git a/docs/docs/get_started/running/k8s.md b/docs/docs/get_started/running/k8s.md new file mode 100644 index 000000000..4259f4554 --- /dev/null +++ b/docs/docs/get_started/running/k8s.md @@ -0,0 +1,81 @@ +--- +description: Deploy Commit-Boost on Kubernetes +--- + +# Kubernetes + +Commit-Boost can be deployed on Kubernetes using the [Helm chart](https://helm.sh/) available in the repository's `provisioning/k8s/commit-boost/` directory. + +## Scope limitation + +:::warning +The current Helm chart supports only the **PBS Service**. It does **not** support the Signer Service or custom commit modules. If you need Signer or module support, please use the [Docker](./docker.md) or [Binary](./binary.md) deployment methods instead. +::: + +## Prerequisites + +- A Kubernetes cluster +- [Helm](https://helm.sh/docs/intro/install/) installed (v3+) + +## Installation + +1. Clone the repository or navigate to the chart directory: + +```bash +git clone https://github.com/Commit-Boost/commit-boost-client.git +cd commit-boost-client/provisioning/k8s/commit-boost +``` + +2. Edit the `values.yaml` file to configure the PBS service according to your needs. The key configuration options are described in the [Values table](#values) below. + +3. Install the chart: + +```bash +helm install commit-boost . -f values.yaml +``` + +This will deploy the Commit-Boost PBS service. By default, the PBS service is available on port `18550`. Point your beacon nodes and validator clients to this port. + +## Values + +The PBS service is configured through the `values.yaml` file. The chart exposes the following key configuration options under the `commitBoost.pbs` section: + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `commitBoost.pbs.enable` | bool | `true` | Enable the PBS service | +| `commitBoost.pbs.image.repository` | string | `ghcr.io/commit-boost/commit-boost` | PBS container image repository | +| `commitBoost.pbs.image.tag` | string | `v0.4.0` | PBS container image tag | +| `commitBoost.pbs.config.chain` | string | `Hoodi` | Ethereum network (e.g. Holesky, Hoodi) | +| `commitBoost.pbs.config.pbs.port` | int | `18550` | PBS service port | +| `commitBoost.pbs.config.relays` | list | `[]` | List of relays to connect to | +| `commitBoost.pbs.config.mux` | list | `[]` | Multiplexer configuration for validator-specific relay routing | +| `commitBoost.pbs.config.metrics.server_port` | int | `10000` | Metrics server port. Note: the config key Commit-Boost actually reads is `metrics.start_port`, so changing `server_port` is currently a no-op (the port stays at the built-in default of `10000`) | +| `replicaCount` | int | `1` | Number of PBS pod replicas | +| `service.type` | string | `ClusterIP` | Kubernetes service type | +| `service.pbs_port` | int | `18550` | Service port for PBS | +| `resources` | object | `{}` | Pod resource requests and limits | +| `autoscaling.enabled` | bool | `false` | Enable horizontal pod autoscaling | + +:::warning +The chart's default `commitBoost.pbs.image.tag` is `v0.4.0`, which is far behind the current release +and predates the unified `commit-boost pbs` CLI the chart invokes. Always set the tag explicitly to +the release you want (e.g. `v0.10.0`) before installing. +::: + +For the full list of available values and their descriptions, see the [README.md](https://github.com/Commit-Boost/commit-boost-client/blob/main/provisioning/k8s/commit-boost/README.md) in the chart directory. + +## Upgrading + +To upgrade an existing release after modifying `values.yaml`: + +```bash +helm upgrade commit-boost . -f values.yaml +``` + +## Uninstalling + +To uninstall the release: + +```bash +helm uninstall commit-boost +``` diff --git a/docs/docs/get_started/running/metrics-catalog.md b/docs/docs/get_started/running/metrics-catalog.md new file mode 100644 index 000000000..957f6f307 --- /dev/null +++ b/docs/docs/get_started/running/metrics-catalog.md @@ -0,0 +1,56 @@ +--- +sidebar_label: "Metrics catalog" +--- + +# Metrics catalog + +Every metric emitted by the Commit-Boost PBS and Signer services, together with the runtime-registered build-info metric from the shared telemetry crate. Useful when building dashboards or writing alerting rules. For scrape and port setup, see [Metrics](./metrics.md). + +--- + +## PBS metrics + +PBS metrics use a custom Prometheus registry with namespace prefix `cb_pbs_`. The registry is created via `Registry::new_custom(Some("cb_pbs"), None)` in `crates/pbs/src/metrics.rs`. All wire names shown below include this prefix. + +| Metric name (wire) | Type | Labels | Description | +|---|---|---|---| +| `cb_pbs_relay_status_code_total` | Counter | `http_status_code`, `endpoint`, `relay_id` | HTTP status code received by relay. Incremented after each relay HTTP response; `http_status_code` may be `"555"` (the value of `TIMEOUT_ERROR_CODE_STR`) for timeouts, or `"556"` (the value of `TRANSPORT_ERROR_CODE`) for WebSocket transport failures on the `get_header` bids stream (unreleased, from v0.11). Endpoint values: `get_header`, `register_validator`, `submit_blinded_block`, `status`. | +| `cb_pbs_relay_latency` | Histogram | `endpoint`, `relay_id` | HTTP latency (duration in seconds) by relay. Records duration of relay HTTP requests. Endpoint values: `get_header`, `register_validator`, `submit_blinded_block`, `status`. | +| `cb_pbs_relay_last_slot` | Gauge | `relay_id` | Latest slot for which a relay delivered a header. Only updated in the `get_header` handler. Set to the current slot on each successful header from that relay. | +| `cb_pbs_relay_header_value` | Gauge | `relay_id` | Header value in gwei delivered by a relay. Converted from raw wei (÷ 1e9) in the `get_header` handler. | +| `cb_pbs_beacon_node_status_code_total` | Counter | `http_status_code`, `endpoint` | HTTP status code returned to the beacon node. Tracks what status codes the PBS returns for beacon node-facing requests. Endpoint values: `get_header`, `register_validator`, `submit_blinded_block`, `status`, `reload`. Error status codes (`502` for `NoResponse`/`NoPayload`, `500` for `Internal`) are set via `PbsClientError`. The handlers also record `406` directly when the request's `Accept` header offers no supported encoding (unreleased, from v0.11 content negotiation), `204` when no bid is available on `get_header`, and `202` for accepted v2 `submit_blinded_block` requests. | +| `cb_pbs_pbs_submit_block_v2_unsupported_total` | Counter | `relay_id` | (unreleased, from v0.11) Count of v2 `submit_blinded_block` requests a relay could not serve because it returned 404 on the v2 endpoint. A non-zero value means the relay does not support `submitBlindedBlockV2` and those blocks were not submitted via that relay. The double `pbs` in the wire name comes from the registry prefix plus the metric name `pbs_submit_block_v2_unsupported_total`. | + +--- + +## Signer metrics + +Signer metrics use a custom Prometheus registry with namespace prefix `cb_signer_`. The registry is created via `Registry::new_custom(Some("cb_signer"), None)` in `crates/signer/src/metrics.rs`. Wire names include this prefix. + +| Metric name (wire) | Type | Labels | Description | +|---|---|---|---| +| `cb_signer_signer_status_code_total` | Counter | `http_status_code`, `endpoint` | HTTP status code returned by signer endpoints. Incremented as responses are sent. Endpoint values: `get_pubkeys`, `generate_proxy_key`, `request_signature_bls`, `request_signature_proxy_bls`, `request_signature_proxy_ecdsa`, and `unknown endpoint` (emitted for the admin routes `/reload` and `/revoke_jwt`, which are matched by the router but not mapped to a named tag). | + +--- + +## Build-info metric (all services) + +When each service starts its metrics HTTP server (via the `MetricsProvider` from the `cb-metrics` crate), a runtime-registered gauge is added to its registry: + +| Metric name (wire) | Type | Labels | Description | +|---|---|---|---| +| `info` | Gauge | `version`, `commit`, `network` | Always `1`. Carries build metadata as Prometheus const labels. The `version` label is the crate version (`CARGO_PKG_VERSION`), `commit` is the Git hash at build time (`GIT_HASH`), and `network` is the chain name (e.g. `Mainnet`, `Holesky`, `Sepolia`, `Hoodi`, or `Custom` for custom chain specs). | + +This metric appears under the service's own registry prefix: the PBS instance exposes it as `cb_pbs_info{version="...",commit="...",network="..."}` and the Signer exposes it as `cb_signer_info{version="...",commit="...",network="..."}`. + +--- + +## Custom module metrics + +Commit modules can register their own metrics via the `prometheus` crate. The module's metrics HTTP server port comes from `CB_METRICS_PORT` (see [Running > Binary](./binary.md#common)). To expose custom metrics: + +1. Create a custom `Registry` (optionally with a namespace prefix). +2. Register your metrics on that registry. +3. Call `MetricsProvider::load_and_run(chain, registry)` to serve the registry on the module's `/metrics` endpoint. Alternatively, construct a `ModuleMetricsConfig` and pass it to `MetricsProvider::new()`, then spawn `provider.run()` yourself. + +All module metrics are served on a separate port and are **not** aggregated into the PBS or Signer registries. To collect them, add the module's metrics port as an additional scrape target in your Prometheus configuration. diff --git a/docs/docs/get_started/running/metrics.md b/docs/docs/get_started/running/metrics.md index 582001959..5b8c96302 100644 --- a/docs/docs/get_started/running/metrics.md +++ b/docs/docs/get_started/running/metrics.md @@ -6,13 +6,15 @@ description: Setup metrics collection Commit-Boost can be configured to collect metrics from the different services and expose them to be scraped from Prometheus. -Make sure to add the `[metrics]` section to your config file: +For a full reference of every metric the PBS and Signer services expose, see the [Metrics catalog](./metrics-catalog.md). + +Make sure to add the `[metrics]` section to your config file (fields and the `start_port` port ladder: [Configuration > Metrics](../configuration.md#metrics)): ```toml [metrics] enabled = true ``` -If the section is missing, metrics collection will be disabled. If you generated the `docker-compose.yml` file with `commit-boost init`, metrics ports will be automatically configured, and a sample `target.json` file will be created. If you're running the binaries directly, you will need to set the correct environment variables, as described in the [previous section](/get_started/running/binary#common). +If you generated the `cb.docker-compose.yml` file with `commit-boost init`, metrics ports will be automatically configured. If you're running the binaries directly, you will need to set the correct environment variables, as described in the [previous section](./binary.md#common). ## Example setup @@ -38,7 +40,7 @@ cb_cadvisor: ### Prometheus -For more information on how to setup Prometheus, see the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/getting_started/). +For more information on how to set up Prometheus, see the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/getting_started/). ```yml title="cb.docker-compose.yml" cb_prometheus: @@ -49,8 +51,13 @@ cb_prometheus: volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml - prometheus-data:/prometheus + networks: + - default + - signer_network ``` +The generated compose file attaches the signer and commit module containers only to `signer_network`, so Prometheus must join that network too or the `cb_signer` and `cb_da_commit` scrape targets below will be unreachable. + ```yml title="prometheus.yml" global: scrape_interval: 15s @@ -62,7 +69,7 @@ scrape_configs: ``` ### Grafana -For more information on how to setup Grafana, see the [Grafana documentation](https://grafana.com/docs/grafana/latest/getting-started/). +For more information on how to set up Grafana, see the [Grafana documentation](https://grafana.com/docs/grafana/latest/fundamentals/getting-started/). ```yml title="cb.docker-compose.yml" cb_grafana: @@ -75,6 +82,8 @@ cb_grafana: - grafana-data:/var/lib/grafana ``` +The mounted datasource provisioning file points Grafana at the Prometheus service: + ```yml title="datasources.yml" apiVersion: 1 @@ -89,6 +98,14 @@ datasources: editable: true ``` -Once Grafana is running, you can [import](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/) the Commit-Boost dashboards from [here](https://github.com/Commit-Boost/commit-boost-client/tree/main/provisioning/grafana), making sure to select the correct `Prometheus` datasource. +### Named volumes +The Prometheus and Grafana services above use the named volumes `prometheus-data` and `grafana-data`. The generated compose file has no top-level `volumes:` block, and Docker Compose refuses to start the whole project if a service references an undeclared volume, so declare both when merging the snippets: + +```yml title="cb.docker-compose.yml" +volumes: + prometheus-data: + grafana-data: +``` +Once Grafana is running, you can [import](https://grafana.com/docs/grafana/latest/visualizations/dashboards/build-dashboards/import-dashboards/) the Commit-Boost dashboards from [here](https://github.com/Commit-Boost/commit-boost-client/tree/main/provisioning/grafana), making sure to select the correct `Prometheus` datasource. diff --git a/docs/docs/get_started/troubleshooting.md b/docs/docs/get_started/troubleshooting.md index bb8623b85..3620d4ae1 100644 --- a/docs/docs/get_started/troubleshooting.md +++ b/docs/docs/get_started/troubleshooting.md @@ -4,18 +4,171 @@ description: Common issues # Troubleshooting -Commit-Boost was recently audited and going through a phased approach for validators to move to production. If you find any or have any question, please reach out on [X (Twitter)](https://x.com/Commit_Boost). If there are any security related items, please see [here](https://github.com/Commit-Boost/commit-boost-client/blob/main/SECURITY.md). +If you run into an issue or have a question, please reach out on [X (Twitter)](https://x.com/Commit_Boost). For security related items, please see [here](https://github.com/Commit-Boost/commit-boost-client/blob/main/SECURITY.md). +--- + +## Where to start + +| If you see… | Likely culprit | Start in section | +|---|---|---| +| Container won't start / exits immediately | Docker (volume, port, image) or missing env vars | [Docker / networking](#docker--networking) | +| HTTP 401 from `POST /signer/*` | JWT auth failure (shared secret mismatch) | [Signer service > JWT auth failures](#jwt-auth-failures) | +| Signer logs a key loading error at startup | Key loading path, format mismatch, or permission error | [Signer service > Key loading](#key-loading) | +| Module log says `connection refused` reaching signer | Docker networking: wrong URL, port, or network | [Docker / networking > container-to-container connectivity](#container-to-container-connectivity) | +| `POST /reload` returns HTTP 500 or 400 | Reload failure: invalid config (500) or bad body override (400) | [Hot reload > Reload failures](#reload-failures) | +| `POST /reload` reverts a previous `POST /revoke_jwt` | Body override not persisted | [Hot reload > Body overrides and footguns](#body-overrides-and-footguns) | +| Module starts but fails all signature requests | Shared secret mismatch, module missing from the signer's config, or clock skew beyond ~5 minutes | [Signer service > JWT auth failures](#jwt-auth-failures) | +| Module container runs but PBS returns no headers | Relays unreachable or timing game expiring too early | [Cascading diagnostics > Scenario 3](#scenario-3-relay-timeout-causes-no-payload-cascade) | +| `docker compose` exits with `no such file` | Missing or misnamed config file or env file | [Docker / networking > Init failures](#init-failures) | + +--- + +## Docker / networking + +### Init failures + +`commit-boost init --config cb-config.toml` produces `cb.docker-compose.yml` and, when the Signer service is enabled, `.cb.env`. If you see `no such file` when running Docker Compose: + +1. Missing config file: verify `cb-config.toml` exists in the working directory and is TOML-valid. +2. Missing env file: if the Signer service is enabled, `.cb.env` is created alongside the compose file. Pass it with `--env-file ./.cb.env`. +3. Wrong path: the volume bindings in the compose file are relative to the working directory. If you moved the config file after `init`, update the `volumes` entry. + +See the [configuration reference](./configuration.md) for a full field listing and [Docker setup](./running/docker.md#init) for init details. + +### Container won't start / exits immediately + +If a container exits immediately after `docker compose up`: + +1. Port conflict: try a different `[pbs] port` or `[signer] port` in the config, or stop whatever is already using the port. A conflict on a published port shows up in the `docker compose up` output as a `port is already allocated` error from the Docker daemon. +2. Missing image: the config's `docker_image` field must point to a valid image. For local development images (e.g. `test_da_commit`), build them first with `just docker-build-test-modules`. +3. Volume mount failure: the config file, keys, and secrets paths must be accessible at runtime. If a path is wrong, the container will exit. A bad config mount logs `Unable to find config file`. +4. Missing environment variables: services that need `CB_CONFIG`, `CB_SIGNER_JWT`, or `CB_MODULE_ID` will fail to start if these aren't set. In the generated compose file, `CB_CONFIG` and `CB_MODULE_ID` are written inline into each service and only the JWT secret comes from `.cb.env` (via `--env-file`); native binaries set all of them on the command line. + +Check `docker compose logs` for the specific error message. + +### Container-to-container connectivity + +Modules connect to the Signer service over an internal Docker bridge network. If a module logs `connection refused`: + +1. Wrong URL: modules receive `CB_SIGNER_URL` as an env var. The default is `http://cb_signer:20000`. If you override this, verify the hostname matches the Signer container name (`cb_signer` by default) and the port matches `[signer] port`. +2. Network isolation: verify the module's compose service is on the `signer_network` (or whatever network the signer is on). The `init` command sets this up automatically; manual compose edits can break it. +3. Signer not healthy: the compose file sets `depends_on: cb_signer: condition: service_healthy`. If the signer fails its health check (e.g., because it can't load keys), dependent modules will never start. Check `docker compose logs cb_signer` first. + +--- + +## Signer service + +If the signer logs an error at startup or signature requests fail at runtime, the likely causes fall into three categories. + +### JWT auth failures + +A `401` response from any `POST /signer/*` endpoint means the request's JWT was rejected. + +1. Shared secret mismatch (most common): each module authenticates with a JWT derived from a shared secret. The signer's `CB_JWTS` env var and the module's `CB_SIGNER_JWT` env var must carry the **same secret for the same module ID**. Common pitfalls: + - Typo in the module ID or secret string. + - The `.cb.env` file was regenerated (e.g., by re-running `init`) but the running containers still use the old env file. + - A manual override was applied via [`POST /reload` body overrides](#body-overrides-and-footguns) but the environment variable was not updated; after a restart the override is lost. +2. Clock skew: the only time-based claim JWT validation checks is `exp` (expiration), with a 10-second leeway. Tokens are minted with a 5-minute expiration, so a token is rejected only when the signer's clock is ahead of the module's by more than the token lifetime plus the leeway, about 5 minutes 10 seconds. A signer clock running behind the module's never invalidates a token. +3. Admin endpoint auth failure: `POST /reload` and `POST /revoke_jwt` require the admin JWT secret (`CB_SIGNER_ADMIN_JWT` environment variable or `admin_secret` body override). If you get a 401 on these endpoints, check that the admin secret matches. + +### Key loading + +If the signer fails to start with errors about keys: + +- The `[signer.local.loader] format` must match the actual keystore layout. See the [Signer configuration](./configuration.md#local-signer) for supported formats and their expected file structures. +- `keys_path` and `secrets_path` are relative to the container's filesystem, not the host. In Docker, these are volume-mounted from the host; verify the mount paths match what the loader expects. +- The signer process runs as a non-root user inside the container, so the mounted keys and secrets must be readable by the container user. +- If `[signer.local.store]` is configured, the proxy directory must exist and be writable. The signer will fail to start if it cannot create proxy key files. +- For Dirk, the remote signer must be reachable at startup. A timeout or connection error during the initial handshake will cause the signer to exit. + +See the [Signer configuration](./configuration.md#signer-service) for a full reference and [Docker setup](./running/docker.md#example-with-pbs-signer-and-a-commit-module) for a working example. + +### TLS + +If you enable TLS and the signer fails to start: + +- The directory set by `path` in `[signer.tls_mode]` (mounted at `/certs` inside the Docker container) is missing `cert.pem` or `key.pem`. +- The key file is not readable by the signer process (a non-root user inside the container). + +See [Configuration > TLS](./configuration.md#tls) for the full certificate contract. + +--- + +## Modules + +### Signer connectivity + +If a commit module logs errors when calling the signer: + +1. Wrong JWT: the module's `CB_SIGNER_JWT` must match the signer's entry for that module ID. See [JWT auth failures](#jwt-auth-failures) above. +2. Wrong signer URL: verify `CB_SIGNER_URL` points to the correct host and port. In Docker, the host is the signer container name (`cb_signer` by default); with native binaries, it is the signer's host IP. +3. Signer not started: modules depend on the signer via Docker Compose `depends_on`. If the signer fails to start (e.g., key loading error), dependent modules will never leave the `created` state. +4. Proxy key errors: generating proxy keys does not require a proxy store; without one the signer only warns that proxies will not be persisted, and they are lost on restart. If `[signer.local.store]` is configured, the directory must be readable and writable or the signer fails at startup (e.g. `failed reading proxy dir: ...`). + +### Module ID mismatch + +If the signer responds with `401` `unauthorized` to a module's requests (or `404` `module id not found` on `POST /revoke_jwt`): + +- The `[[modules]]` entry in `cb-config.toml` uses a different `id` than what the module was started with (`CB_MODULE_ID` env var). These must match exactly. +- After adding a new module to the config, send [`POST /reload`](#hot-reload) to the signer before starting the module container. Until then, the signer has no record of the new module and will reject its requests. `/reload` returns `500` with `JWT secret for module X is missing` if the secret was not in `CB_JWTS` at signer startup; update `.cb.env` and restart. Full mechanism: [Hot reload > Common patterns](./configuration.md#common-patterns). + +--- + +## Hot reload + +Commit-Boost supports hot-reloading the configuration without restarting containers. The mechanism is fully documented in the [configuration page](./configuration.md#hot-reload); this section covers what to do when it breaks. + +### Reload failures + +`500` means the reload was rejected and the previous configuration kept: usually invalid TOML in the changed config file (check `docker compose logs` for the parse error), sometimes a permission error re-reading it. `400` means a body override references a module ID that is not in the config file. Full rules: [Hot reload](./configuration.md#hot-reload). + +### Body overrides and footguns + +The `POST /reload` body overrides (`jwt_secrets`, `admin_secret`) live only in memory and are lost on restart, and a `/reload` re-adds any module revoked with `POST /revoke_jwt` that is still in the config. Full rules: [Signer service reload](./configuration.md#signer-service-reload). + +### Hot reload and custom PBS + +Custom PBS services may override the default reload behavior to parse extra configuration fields. If a custom PBS returns `500` on reload, check the module's documentation for custom reload handling. See the [custom module examples](https://github.com/Commit-Boost/commit-boost-client/blob/main/examples/status_api/src/main.rs) for details. + +--- + +## Cascading diagnostics + +Failures in one service often propagate to others. When debugging, check the upstream dependency first. + +### Scenario 1: Signer fails to load keys, all modules fail + +A signer that cannot read its keystore fails its health check, so Docker Compose never marks `cb_signer` as healthy and modules guarded by `depends_on: condition: service_healthy` never start. Anything that needs proposer commitments (proxy key generation, signature requests) then gets `connection refused`. + +Start with the signer log. A key loading error at the top (e.g. `failed reading proxy dir: ...` or a keystore parse failure) means all downstream failures are consequences. Fix the key loading, then restart. + +### Scenario 2: Config file becomes stale after a restart + +Telltale pattern: everything worked until a restart, then all modules fail with 401. Cause: a body-override secret rotation was never persisted (see [Body overrides and footguns](#body-overrides-and-footguns)). Update `.cb.env` to the rotated secret and restart. + +### Scenario 3: Relay timeout causes no-payload cascade + +When a relay becomes slow or unresponsive, PBS times out waiting for that relay's header. If no relay returns a valid bid, PBS returns `204` (no content) to the CL, which falls back to the local execution payload, so there is no MEV reward. If the request itself fails (rather than simply yielding no bids), PBS instead returns `502` (`no payload from relays`). + +Check the PBS logs for a specific relay repeatedly timing out: a `get_header` timeout logs `err="Timed Out"` with the relay id, and increments `cb_pbs_relay_status_code_total` with `http_status_code="555"` (the synthetic timeout code; see the [Metrics catalog](./running/metrics-catalog.md)). Remove or replace that relay in the `[[relays]]` config, then `POST /reload` the PBS. + +--- + +## Expected healthy logs If you started the modules correctly you should see the following logs. -## PBS -After the module started correctly you should see: -```bash -2024-09-16T19:27:16.004643Z INFO Starting PBS service address=0.0.0.0:18550 events_subs=0 +### PBS + +After the service started correctly you should see: +```text +2025-11-04T14:22:03.118512Z INFO starting PBS service version="0.10.0-rc4" commit_hash="eeff25750c01f4adfc95fc08d69d541ace8e4087" addr=0.0.0.0:18550 chain=Hoodi ``` -To check that the setup is correct and you are connected to relays, you can trigger manually the `/status` endpoint, by running: +The v0.10.0 release commit self-reports version `0.10.0-rc4`; any other checkout prints its own commit hash and version. + +To check that the setup is correct and you are connected to relays, you can manually trigger the `/status` endpoint, by running: ```bash curl http://0.0.0.0:18550/eth/v1/builder/status -vvv @@ -30,39 +183,48 @@ curl http://0.0.0.0:18550/eth/v1/builder/status -vvv * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < content-length: 0 -< date: Mon, 16 Sep 2024 19:32:07 GMT +< date: Tue, 04 Nov 2025 14:32:07 GMT < * Connection #0 to host 0.0.0.0 left intact ``` -if now you check the logs, you should see: +If you now check the logs, you should see: -```bash -2024-09-16T19:32:07.634966Z INFO status{req_id=62f1c0db-f277-49fa-91e7-a9a1c2b2a6d3}: ua="curl/7.81.0" relay_check=true -2024-09-16T19:32:07.642992Z INFO status{req_id=62f1c0db-f277-49fa-91e7-a9a1c2b2a6d3}: relay check successful +```text +2025-11-04T14:32:07.634966Z INFO : new request ua="curl/7.81.0" relay_check=true method=/eth/v1/builder/status req_id=62f1c0db-f277-49fa-91e7-a9a1c2b2a6d3 +2025-11-04T14:32:07.642992Z INFO : relay check successful method=/eth/v1/builder/status req_id=62f1c0db-f277-49fa-91e7-a9a1c2b2a6d3 +2025-11-04T14:32:07.643104Z INFO : Responded with 200 OK in 8 ms method=/eth/v1/builder/status req_id=62f1c0db-f277-49fa-91e7-a9a1c2b2a6d3 ``` -If the sidecar is setup correctly, it will receive and process calls from the CL: +The leading `:` is the (deliberately unnamed) request span; the fields after the message are that +span's fields, so every line belonging to one request carries the same `req_id`. + +If the sidecar is set up correctly, it will receive and process calls from the CL: + #### Register validator This should happen periodically, depending on your validator setup. -```bash -2024-09-16T19:28:37.976534Z INFO register_validators{req_id=296f662f-0e7a-4f15-be75-55b8ca19ffc0}: ua="Lighthouse/v5.2.1-9e12c21" num_registrations=500 -2024-09-16T19:28:38.819591Z INFO register_validators{req_id=296f662f-0e7a-4f15-be75-55b8ca19ffc0}: register validator successful +```text +2025-11-04T14:28:37.976534Z INFO : new request ua="Lighthouse/v5.2.1-9e12c21" num_registrations=500 method=/eth/v1/builder/validators req_id=296f662f-0e7a-4f15-be75-55b8ca19ffc0 +2025-11-04T14:28:38.819591Z INFO : register validator successful method=/eth/v1/builder/validators req_id=296f662f-0e7a-4f15-be75-55b8ca19ffc0 ``` #### Get header This will only happen if some of your validators have a proposal slot coming up. -```bash -2024-09-16T19:30:24.135376Z INFO get_header{req_id=74126c5f-69e6-4961-86a6-6c2597bf15f5 slot=2551052}: ua="Lighthouse/v5.2.1-9e12c21" parent_hash=0x641c99d6e4f14bf6d268eb2a8c0dc51c7030ab24e384c0e679f2a6b438d298ea validator_pubkey=0x84fc20b09496341f24abfcb6f407e916ecc317497c5b1bba4970e50e96cf5e731b88e51753064c30cb221453bd71aebf ms_into_slot=135 -2024-09-16T19:30:25.089477Z INFO get_header{req_id=74126c5f-69e6-4961-86a6-6c2597bf15f5 slot=2551052}: received header block_hash=0x0139686e8d251f010153875270256fce6f298d7b3f3f9129179fb86297dffad3 value_eth="0.001399518501462470" +```text +2025-11-04T14:30:24.135376Z INFO : new request ua="Lighthouse/v5.2.1-9e12c21" ms_into_slot=135 method=/eth/v1/builder/header/{slot}/{parent_hash}/{pubkey} req_id=74126c5f-69e6-4961-86a6-6c2597bf15f5 slot=1671102 parent_hash=0x641c99d6e4f14bf6d268eb2a8c0dc51c7030ab24e384c0e679f2a6b438d298ea validator=0x84fc20b09496341f24abfcb6f407e916ecc317497c5b1bba4970e50e96cf5e731b88e51753064c30cb221453bd71aebf +2025-11-04T14:30:25.089477Z INFO : received header value_eth="0.001399518501462470" block_hash=0x0139686e8d251f010153875270256fce6f298d7b3f3f9129179fb86297dffad3 method=/eth/v1/builder/header/{slot}/{parent_hash}/{pubkey} req_id=74126c5f-69e6-4961-86a6-6c2597bf15f5 slot=1671102 parent_hash=0x641c99d6e4f14bf6d268eb2a8c0dc51c7030ab24e384c0e679f2a6b438d298ea validator=0x84fc20b09496341f24abfcb6f407e916ecc317497c5b1bba4970e50e96cf5e731b88e51753064c30cb221453bd71aebf ``` +If no relay returns a usable bid you will see `no header available for slot` instead, and PBS answers the beacon node with a `204`. + #### Submit block This will only happen if you received a header in the previous call, and if the header is higher than the locally built block. -```bash -2024-09-16T14:38:01.409075Z INFO submit_blinded_block{req_id=6eb9a04d-6f79-4295-823f-c054582b3599 slot=2549590}: ua="Lighthouse/v5.2.1-9e12c21" slot_uuid=16186e06-0cd0-47bc-9758-daa1b66eff5c ms_into_slot=1409 block_hash=0xfa135ae6f2bfb32b0a47368f93d69e0a2b3f8b855d917ec61d78e78779edaae6 -2024-09-16T14:38:02.910974Z INFO submit_blinded_block{req_id=6eb9a04d-6f79-4295-823f-c054582b3599 slot=2549590}: received unblinded block +```text +2025-11-04T14:30:26.409075Z INFO : new request ua="Lighthouse/v5.2.1-9e12c21" ms_into_slot=2409 method=/eth/v1/builder/blinded_blocks req_id=6eb9a04d-6f79-4295-823f-c054582b3599 slot=1671102 block_hash=0x0139686e8d251f010153875270256fce6f298d7b3f3f9129179fb86297dffad3 block_number=1640231 parent_hash=0x641c99d6e4f14bf6d268eb2a8c0dc51c7030ab24e384c0e679f2a6b438d298ea +2025-11-04T14:30:27.910974Z INFO : received unblinded block (v1) method=/eth/v1/builder/blinded_blocks req_id=6eb9a04d-6f79-4295-823f-c054582b3599 slot=1671102 block_hash=0x0139686e8d251f010153875270256fce6f298d7b3f3f9129179fb86297dffad3 block_number=1640231 parent_hash=0x641c99d6e4f14bf6d268eb2a8c0dc51c7030ab24e384c0e679f2a6b438d298ea ``` + +A beacon node calling the v2 route (`POST /eth/v2/builder/blinded_blocks`) logs `received unblinded block (v2)` instead. diff --git a/docs/docs/overview.md b/docs/docs/overview.md index 5005bda89..0a54a0df5 100644 --- a/docs/docs/overview.md +++ b/docs/docs/overview.md @@ -2,27 +2,25 @@ sidebar_position: 2 --- -# Overview +# Why Commit-Boost ## Background -- Proposer commitments have been an important part of Ethereum’s history. Today, we already see the power of commitments where over 90% of validators give up their autonomy and make a wholesale commitment that outsources block building to a sophisticated actor called a block builder. -- However, most are starting to agree on a common denominator: in the future, beacon proposers will face a broader set of options of what they may “commit" to–be it inclusions lists or preconfs or other types of commitments such as long-dated blockspace futures–compared to just an external or local payload they see today. -- A recent post from Barnabe captures this well; during block construction, the validator “…creates the specs, or the template, by which the resulting block must be created, and the builders engaged by the proposer are tasked with delivering the block according to its specifications”. -- While this all seems great, the challenge is that many teams building commitments are creating new sidecars driving fragmentation and risks for Ethereum. -- For Ethereum, there are going to be significant challenges and increased risks during upgrades if there are a handful of sidecars validators are running. -- For validators, these risks potentially take us to a world where proposers will need to make decisions on which teams to “bet on” and which sidecars they will need to run to participate in what those teams are offering. -- For homestakers, this is difficult and they likely will be unable to participate in more than one of these commitments. -- For sophisticated actors, this increases the attack vector and operational complexity as more and more sidecars are required to be run. -- Another side effect of this is validators are somewhat locked into using a specific sidecar due to limited operational capacity and the switching costs of running a different sidecar (i.e., vendor lock-in). The higher the switching costs, the more embedded network effects could become if these sidecars only support certain downstream actors / proposer commitment protocols. -- This also could create a dynamic where core out-of-protocol infrastructure supporting Ethereum which should be a public good, starts being used for monetization, distribution, or other purposes. -- Commit-Boost aims to standardize how proposer commitment protocols communicate with the proposer, by providing a unified interface implemented in a single validator sidecar with the goal of reducing fragmentation. +Proposer commitments have been an important part of Ethereum’s history. Today, we already see the power of commitments where over 90% of validators give up their autonomy and make a wholesale commitment that outsources block building to a sophisticated actor called a block builder. + +Most are starting to agree on a common denominator: in the future, beacon proposers will face a broader set of options of what they may “commit” to (be it inclusion lists, preconfs, or other types of commitments such as long-dated blockspace futures) compared to just an external or local payload they see today. A recent post from Barnabe captures this well; during block construction, the validator “…creates the specs, or the template, by which the resulting block must be created, and the builders engaged by the proposer are tasked with delivering the block according to its specifications”. + +The challenge is that many teams building commitments are creating new sidecars, driving fragmentation and risks for Ethereum. For Ethereum, there are going to be significant challenges and increased risks during upgrades if there are a handful of sidecars validators are running. For validators, these risks potentially take us to a world where proposers will need to make decisions on which teams to “bet on” and which sidecars they will need to run to participate in what those teams are offering. For homestakers, this is difficult and they likely will be unable to participate in more than one of these commitments. For sophisticated actors, this increases the attack vector and operational complexity as more and more sidecars are required to be run. + +Another side effect is that validators are somewhat locked into using a specific sidecar due to limited operational capacity and the switching costs of running a different sidecar (i.e., vendor lock-in). The higher the switching costs, the more embedded network effects could become if these sidecars only support certain downstream actors / proposer commitment protocols. This also could create a dynamic where core out-of-protocol infrastructure supporting Ethereum, which should be a public good, starts being used for monetization, distribution, or other purposes. + +Commit-Boost's answer is a unified interface implemented in a single validator sidecar. ## Goals - Unify behind a software / standard to reduce fragmentation risks for Ethereum and its validators, while ensuring open innovation downstream from the proposer can flourish. - Create a neutral, open-source, public good for the safe development and distribution of proposer commitments protocols. - Provide a well-tested, reliable validator sidecar with support for advanced observability and telemetry. -## Why Commit-Boost? +## Who it serves ### For validators - Run a single sidecar with support for MEV-Boost and other proposer commitments protocols, such as preconfirmations and inclusion lists. diff --git a/docs/docs/res/img/consensus-key-sign.png b/docs/docs/res/img/consensus-key-sign.png new file mode 100644 index 000000000..118903f68 Binary files /dev/null and b/docs/docs/res/img/consensus-key-sign.png differ diff --git a/docs/docs/res/img/proxy-key-sign.png b/docs/docs/res/img/proxy-key-sign.png new file mode 100644 index 000000000..e665d24cf Binary files /dev/null and b/docs/docs/res/img/proxy-key-sign.png differ diff --git a/docs/sidebars.js b/docs/sidebars.js index 7b3fc68d2..3fcef6cfc 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -29,7 +29,9 @@ const sidebars = { collapsed: false, items: [ 'get_started/overview', + 'get_started/building', 'get_started/configuration', + 'get_started/mux-key-loaders', { type: 'category', label: 'Running', @@ -39,8 +41,9 @@ const sidebars = { items: [ 'get_started/running/docker', 'get_started/running/binary', + 'get_started/running/k8s', 'get_started/running/metrics', - + 'get_started/running/metrics-catalog', ], }, 'get_started/troubleshooting', @@ -53,8 +56,9 @@ const sidebars = { type: 'generated-index', }, items: [ - 'developing/custom-modules', - 'developing/commit-module', + 'developing/commit-modules', + 'developing/prop-commit-signing', + 'developing/extending-pbs', 'developing/environment-setup', ], },