From ba7639a4570d33d0c958fb4e27908c383e06a84d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1s=20Gr=C3=BCner?= <47506558+MegaRedHand@users.noreply.github.com> Date: Tue, 11 Aug 2026 17:10:58 -0300 Subject: [PATCH 1/7] docs: slots and intervals --- docs/slots_and_intervals.md | 39 +++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 docs/slots_and_intervals.md diff --git a/docs/slots_and_intervals.md b/docs/slots_and_intervals.md new file mode 100644 index 00000000..74b3c398 --- /dev/null +++ b/docs/slots_and_intervals.md @@ -0,0 +1,39 @@ +# Slots and Intervals + +A Lean Chain slot has a duration of 4 seconds and is divided in 5 intervals: + +1. Block proposal +2. Vote propagation +3. Vote aggregation +4. Safe target computation +5. Head update + +```text + ONE SLOT (4000 ms) + ┌────────────┬────────────┬────────────┬────────────┬────────────┐ + │ Interval 0 │ Interval 1 │ Interval 2 │ Interval 3 │ Interval 4 │ + │ t+0 ms │ t+800 ms │ t+1600 ms │ t+2400 ms │ t+3200 ms │ + ├────────────┼────────────┼────────────┼────────────┼────────────┤ + │ block │ vote │ vote │ safe target│ head │ + │ proposal │propagation │aggregation │computation │ update │ + └────────────┴────────────┴────────────┴────────────┴────────────┘ +``` + +Block proposal is the first interval of a slot. During this interval, a block proposer, selected in a round-robin fashion, proposes a new block and gossips it to the network. Right before building the block, the proposer merges their "new attestations buffer" into their fork-choice view. They then include attestations that the proposer has recently seen into their block. Other validators verify the block and its contents, and merge the votes it includes into their fork-choice view. After importing a block, all validators recompute their [head](./lmd_ghost.md), and update the latest [finalized and justified checkpoints](./3sf_mini.md) according to the block's post-state. + +Vote propagation is the second interval of a slot. During this interval, validators gossip their votes for the block they consider to be the head of the chain, and append to it a `(source, target)` [finality vote](./3sf_mini.md). These votes are in aggregation subnets and are imported by aggregators. Aggregators verify the votes in their subnet and store them for later aggregation. + +Vote aggregation is the third interval of a slot. During this interval, aggregators aggregate the votes they have received and gossip the resulting aggregated attestations to the network. These aggregated attestations are imported by all validators, who verify and store them in a "new attestations buffer". + +Safe target computation is the fourth interval of a slot. During this interval, validators compute the [safe target](./lmd_ghost.md#safe-target-selection) they'll use when deciding which finality vote to cast on the next slot. The safe target is computed based on the votes received in the current slot. + +Head update is the fifth and final interval of a slot. During this interval, validators merge the aggregated attestations they have in their "new attestations buffer" into their fork-choice view, and recompute their head. + +> **In ethlambda:** the intervals are the `SlotInterval` variants in +> `crates/blockchain/src/lib.rs`, and their length comes from +> `MILLISECONDS_PER_INTERVAL` and `INTERVALS_PER_SLOT` in +> `crates/common/types/src/constants.rs`. Block proposal is merged into the +> previous slot's head-update interval: the proposer advances its store to the +> next slot, builds the block there, and holds publication until the slot +> boundary. That buys the build one extra interval of headroom and leaves no +> actor work at the block-proposal tick itself. From ba5431c750ca9de775dd169b65f55107ee097d91 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1s=20Gr=C3=BCner?= <47506558+MegaRedHand@users.noreply.github.com> Date: Tue, 11 Aug 2026 17:11:29 -0300 Subject: [PATCH 2/7] docs: link new doc and update lmd-ghost --- docs/SUMMARY.md | 1 + docs/introduction.md | 11 +++++---- docs/lmd_ghost.md | 56 ++++++++++++++++++++++++-------------------- 3 files changed, 38 insertions(+), 30 deletions(-) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 2c059f13..a7ac16fa 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -4,6 +4,7 @@ # Consensus +- [Slots and Intervals](./slots_and_intervals.md) - [3SF-mini: Justification & Finalization](./3sf_mini.md) - [LMD-GHOST Fork Choice](./lmd_ghost.md) diff --git a/docs/introduction.md b/docs/introduction.md index 06e5bc7c..4ffe4caf 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -6,11 +6,12 @@ consensus client, written in Rust. This book collects the design notes and operator-facing references for ethlambda. It is split into two parts: -- **Consensus** explains the algorithms ethlambda implements: the - [3SF-mini](./3sf_mini.md) justification and finalization rules, and the - [LMD-GHOST](./lmd_ghost.md) fork choice algorithm. Both documents are - implementation-agnostic; ethlambda-specific behaviour is called out in - blockquotes. +- **Consensus** explains how the chain advances: the + [slot and interval structure](./slots_and_intervals.md) that schedules every + validator duty, the [3SF-mini](./3sf_mini.md) justification and finalization + rules, and the [LMD-GHOST](./lmd_ghost.md) fork choice algorithm. These + documents are implementation-agnostic; ethlambda-specific behaviour is called + out in blockquotes. - **Operations** documents observable surfaces of a running node: [Prometheus metrics](./metrics.md), [checkpoint sync](./checkpoint_sync.md), and the [fork choice visualization](./fork_choice_visualization.md) served diff --git a/docs/lmd_ghost.md b/docs/lmd_ghost.md index 980fd3c1..38cca6cd 100644 --- a/docs/lmd_ghost.md +++ b/docs/lmd_ghost.md @@ -455,9 +455,9 @@ designated moments. This ensures all validators operate on a consistent view. fixed points ``` -> **In ethlambda:** The two stages are called "new" and "known" attestations, stored -> in `LatestNewAttestations` and `LatestKnownAttestations` tables respectively. -> Promotion happens at tick intervals 0 (if proposing) and 3 (end of slot). +> **In ethlambda:** The two stages are called "new" and "known" attestations, held in +> the in-memory `new_payloads` and `known_payloads` buffers of the `Store` respectively. +> Promotion happens at tick intervals 0 (if proposing) and 4 (end of slot). ### Why Staged Promotion? @@ -618,27 +618,28 @@ source code locations, and performance. ### Tick-Based Scheduling -ethlambda divides time into **4-second slots**, each split into **4 intervals** (1 second -each). Fork choice operations are scheduled at specific intervals: +ethlambda divides time into **4-second slots**, each split into **5 intervals** (800 ms +each), as described in [Slots and Intervals](./slots_and_intervals.md). Fork choice +operations are scheduled at specific intervals: ```text - ONE SLOT (4 seconds) - ┌──────────────┬──────────────┬──────────────┬──────────────┐ - │ Interval 0 │ Interval 1 │ Interval 2 │ Interval 3 │ - │ (t+0s) │ (t+1s) │ (t+2s) │ (t+3s) │ - ├──────────────┼──────────────┼──────────────┼──────────────┤ - │ │ │ │ │ - │ IF PROPOSER: │ NON-PROPOSER:│ update_safe │ accept_new │ - │ accept new │ produce │ _target() │ _attestations│ - │ attestations│ attestation │ │ () │ - │ + propose │ │ (2/3 vote │ │ - │ block │ │ threshold) │ update_head()│ - │ │ │ │ │ - │ update_head()│ │ │ │ - │ │ │ │ │ - └──────────────┴──────────────┴──────────────┴──────────────┘ - - ◄─────────────── Slot N ──────────────────────────────────────► + ONE SLOT (4000 ms) + ┌────────────┬────────────┬────────────┬────────────┬────────────┐ + │ Interval 0 │ Interval 1 │ Interval 2 │ Interval 3 │ Interval 4 │ + │ t+0 ms │ t+800 ms │ t+1600 ms │ t+2400 ms │ t+3200 ms │ + ├────────────┼────────────┼────────────┼────────────┼────────────┤ + │ │ │ │ │ │ + │IF PROPOSER:│ ALL │ aggregators│update_safe │accept_new_ │ + │ accept new │ VALIDATORS:│ publish │_target() │attestations│ + │ attestation│ produce │ aggregated │ │() │ + │ + propose │ attestation│ attestation│ (2/3 vote │ │ + │ block │ │ │ threshold) │update_head │ + │ │ │ │ │() │ + │update_head │ │ │ │ │ + │() │ │ │ │ │ + └────────────┴────────────┴────────────┴────────────┴────────────┘ + + ◄─────────────── Slot N ──────────────────────────────────────────► ``` **Detailed sequence:** @@ -655,20 +656,25 @@ each). Fork choice operations are scheduled at specific intervals: │ Interval 1 ─ Attestation production │ - ├── Non-proposers: + ├── All validators, proposer included: │ └── Create attestation with: │ • head = current fork choice head (newest head) │ • target = derived from safe_target (for 3SF-mini) │ • source = latest_justified checkpoint │ Publish attestation to gossipsub │ - Interval 2 ─ Safe target update + Interval 2 ─ Aggregation + │ + ├── Aggregators: aggregate their subnet's gossip signatures + │ └── Publish the aggregated attestation to gossipsub + │ + Interval 3 ─ Safe target update │ ├── Recalculate safe_target using 2/3 supermajority threshold │ └── Only blocks with ≥ ⌈2V/3⌉ attestation weight qualify │ (V = total validators) │ - Interval 3 ─ End of slot + Interval 4 ─ End of slot │ ├── Promote new → known attestations └── Run fork choice → update_head() From f56be4205f614fe135bb8a2f25bddc0ba9ac5c5d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1s=20Gr=C3=BCner?= <47506558+MegaRedHand@users.noreply.github.com> Date: Tue, 11 Aug 2026 17:29:42 -0300 Subject: [PATCH 3/7] docs: do an improvement pass --- docs/slots_and_intervals.md | 111 +++++++++++++++++++++++++++++------- 1 file changed, 91 insertions(+), 20 deletions(-) diff --git a/docs/slots_and_intervals.md b/docs/slots_and_intervals.md index 74b3c398..4a6f1ecb 100644 --- a/docs/slots_and_intervals.md +++ b/docs/slots_and_intervals.md @@ -1,12 +1,15 @@ # Slots and Intervals -A Lean Chain slot has a duration of 4 seconds and is divided in 5 intervals: +A Lean Chain slot has a duration of 4 seconds and is divided in 5 intervals of 800 ms. +Every duty a validator owes the chain is due in one of them: -1. Block proposal -2. Vote propagation -3. Vote aggregation -4. Safe target computation -5. Head update +| Interval | Offset | Duty | Who acts | What it publishes | +| --- | --- | --- | --- | --- | +| 0 | t+0 ms | [Block proposal](#interval-0-block-proposal) | the slot's proposer | the block, on the `block` topic | +| 1 | t+800 ms | [Vote propagation](#interval-1-vote-propagation) | every validator | a signed attestation, on its subnet topic | +| 2 | t+1600 ms | [Vote aggregation](#interval-2-vote-aggregation) | aggregators | an aggregated attestation, on the `aggregation` topic | +| 3 | t+2400 ms | [Safe target computation](#interval-3-safe-target-computation) | every validator | nothing: local bookkeeping | +| 4 | t+3200 ms | [Head update](#interval-4-head-update) | every validator | nothing: local bookkeeping | ```text ONE SLOT (4000 ms) @@ -14,26 +17,94 @@ A Lean Chain slot has a duration of 4 seconds and is divided in 5 intervals: │ Interval 0 │ Interval 1 │ Interval 2 │ Interval 3 │ Interval 4 │ │ t+0 ms │ t+800 ms │ t+1600 ms │ t+2400 ms │ t+3200 ms │ ├────────────┼────────────┼────────────┼────────────┼────────────┤ - │ block │ vote │ vote │ safe target│ head │ + │ block │ vote │ vote │safe target │ head │ │ proposal │propagation │aggregation │computation │ update │ └────────────┴────────────┴────────────┴────────────┴────────────┘ + ◄───────────── gossiped ─────────────▶ ◄───── local only ───────▶ ``` -Block proposal is the first interval of a slot. During this interval, a block proposer, selected in a round-robin fashion, proposes a new block and gossips it to the network. Right before building the block, the proposer merges their "new attestations buffer" into their fork-choice view. They then include attestations that the proposer has recently seen into their block. Other validators verify the block and its contents, and merge the votes it includes into their fork-choice view. After importing a block, all validators recompute their [head](./lmd_ghost.md), and update the latest [finalized and justified checkpoints](./3sf_mini.md) according to the block's post-state. +The grid comes from a genesis timestamp every node shares, so the schedule needs no +coordination messages: a node reads its clock, works out which interval it is in, and +knows which duty is due. The order is a dependency chain, since each interval consumes +what the previous one produced. A duty that overruns its interval is not rescheduled: it +lands late, and the slot moves on without it. -Vote propagation is the second interval of a slot. During this interval, validators gossip their votes for the block they consider to be the head of the chain, and append to it a `(source, target)` [finality vote](./3sf_mini.md). These votes are in aggregation subnets and are imported by aggregators. Aggregators verify the votes in their subnet and store them for later aggregation. +> **In ethlambda:** the intervals are the `SlotInterval` variants in +> `crates/blockchain/src/lib.rs`, and their length comes from +> `MILLISECONDS_PER_INTERVAL` and `INTERVALS_PER_SLOT` in +> `crates/common/types/src/constants.rs`. -Vote aggregation is the third interval of a slot. During this interval, aggregators aggregate the votes they have received and gossip the resulting aggregated attestations to the network. These aggregated attestations are imported by all validators, who verify and store them in a "new attestations buffer". +## Interval 0: Block proposal -Safe target computation is the fourth interval of a slot. During this interval, validators compute the [safe target](./lmd_ghost.md#safe-target-selection) they'll use when deciding which finality vote to cast on the next slot. The safe target is computed based on the votes received in the current slot. +A block proposer, selected in a round-robin fashion (`slot % num_validators`), proposes a +new block and gossips it to the network. Right before building the block, the proposer +merges their "new attestations buffer" into their fork-choice view. They then include +attestations that the proposer has recently seen into their block. Other validators verify +the block and its contents, and merge the votes it includes into their fork-choice view. +After importing a block, all validators [recompute their head](./lmd_ghost.md), and update +the latest [finalized and justified checkpoints](./3sf_mini.md) according to the block's +post-state. -Head update is the fifth and final interval of a slot. During this interval, validators merge the aggregated attestations they have in their "new attestations buffer" into their fork-choice view, and recompute their head. +A block body carries at most +`MAX_ATTESTATIONS_DATA` aggregated attestations: distinct `(slot, head, target, source)` tuples, each paired with a +bitfield naming the validators bound to it. +Genesis occupies slot 0, so proposals start at slot 1, and nothing forces a slot to be +filled: a proposer that is offline or too slow leaves an empty slot, and the next block +simply points its parent root at an older block. -> **In ethlambda:** the intervals are the `SlotInterval` variants in -> `crates/blockchain/src/lib.rs`, and their length comes from -> `MILLISECONDS_PER_INTERVAL` and `INTERVALS_PER_SLOT` in -> `crates/common/types/src/constants.rs`. Block proposal is merged into the -> previous slot's head-update interval: the proposer advances its store to the -> next slot, builds the block there, and holds publication until the slot -> boundary. That buys the build one extra interval of headroom and leaves no -> actor work at the block-proposal tick itself. +> **In ethlambda:** block proposal is merged into the previous slot's head-update +> interval: the proposer advances its store to the next slot, builds the block there, +> and holds publication until the slot boundary. That buys the build one extra interval +> of headroom and leaves no actor work at the block-proposal tick itself. + +## Interval 1: Vote propagation + +Validators gossip their votes for the block they consider to be the head of the chain, and +append to it a `(source, target)` [finality vote](./3sf_mini.md#recap-attestation-anatomy). +These votes are in aggregation subnets and are imported by aggregators. Aggregators verify +the votes in their subnet and store them for later aggregation. + +> **In ethlambda:** a validator's subnet is `validator_index % attestation_committee_count`, +> and a node only aggregates for subnets it subscribed to at startup. Aggregation is also +> gated on the aggregator role, seeded by `--is-aggregator` and flippable at runtime +> through the admin API. A chain whose validators all decline the role still gossips votes +> and logs them as processed, but no aggregate is ever produced, so every block is empty +> and the chain never justifies. + +## Interval 2: Vote aggregation + +Aggregators aggregate the votes they have received and gossip the resulting aggregated +attestations to the network. These aggregated attestations are imported by all validators, +who verify and store them in a "new attestations buffer". + +Aggregation earns its own interval because it is the heaviest recurring computation in the +client: collapsing a subnet's worth of them +into one proof is heavy CPU work. It is also what makes a block affordable, since a block +carrying raw votes would need one full XMSS signature per voter, quickly going over the network bandwidth limit. + +> **In ethlambda:** the proofs run on an off-thread worker so the blockchain actor's +> message loop stays responsive, and a session may start up to `EARLY_AGGREGATION_WINDOW` +> before the interval boundary once two thirds of the signatures are in. At the session's +> soft deadline the actor stops handing out new jobs, but a proof already in flight +> finishes and publishes late rather than being discarded. + +## Interval 3: Safe target computation + +Validators compute the [safe target](./lmd_ghost.md#safe-target-selection) they'll use when +deciding which finality vote to cast on the next slot. The safe target is computed based on +the votes received in the current slot. + +It is LMD-GHOST run with a two-thirds weight threshold instead of a plain majority, so it +sits at or behind the head and advances only once a branch is backed by a supermajority. +Deriving targets from it is what stops [3SF-mini](./3sf_mini.md) from justifying a branch +the network has not visibly converged on. + +## Interval 4: Head update + +Validators merge the aggregated attestations they have in their "new attestations buffer" +into their fork-choice view, and recompute their head. + +This is the slot's second and last promotion point; the first is the proposer's, just +before it builds. Between promotions a vote sits in the buffer without weight, which is +what keeps a validator's fork-choice view from shifting under it mid-slot. See +[why staged promotion](./lmd_ghost.md#why-staged-promotion) for the reasoning. From 3d452f5513f698ced50395dc8ea437d08750bb8c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1s=20Gr=C3=BCner?= <47506558+MegaRedHand@users.noreply.github.com> Date: Tue, 11 Aug 2026 17:42:47 -0300 Subject: [PATCH 4/7] docs: second round of improvements --- docs/slots_and_intervals.md | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/docs/slots_and_intervals.md b/docs/slots_and_intervals.md index 4a6f1ecb..93b2bcee 100644 --- a/docs/slots_and_intervals.md +++ b/docs/slots_and_intervals.md @@ -45,9 +45,9 @@ After importing a block, all validators [recompute their head](./lmd_ghost.md), the latest [finalized and justified checkpoints](./3sf_mini.md) according to the block's post-state. -A block body carries at most -`MAX_ATTESTATIONS_DATA` aggregated attestations: distinct `(slot, head, target, source)` tuples, each paired with a -bitfield naming the validators bound to it. +A block body carries at most `MAX_ATTESTATIONS_DATA` aggregated attestations: distinct +`(slot, head, target, source)` tuples, each paired with a bitfield naming the validators +bound to it. Genesis occupies slot 0, so proposals start at slot 1, and nothing forces a slot to be filled: a proposer that is offline or too slow leaves an empty slot, and the next block simply points its parent root at an older block. @@ -77,10 +77,10 @@ Aggregators aggregate the votes they have received and gossip the resulting aggr attestations to the network. These aggregated attestations are imported by all validators, who verify and store them in a "new attestations buffer". -Aggregation earns its own interval because it is the heaviest recurring computation in the -client: collapsing a subnet's worth of them -into one proof is heavy CPU work. It is also what makes a block affordable, since a block -carrying raw votes would need one full XMSS signature per voter, quickly going over the network bandwidth limit. +Aggregation earns its own interval because collapsing a subnet's worth of XMSS signatures +into one proof is the heaviest recurring computation in the client. It is also what makes a +block affordable, since a block carrying raw votes would need one full XMSS signature per +voter, quickly going over the network bandwidth limit. > **In ethlambda:** the proofs run on an off-thread worker so the blockchain actor's > message loop stays responsive, and a session may start up to `EARLY_AGGREGATION_WINDOW` @@ -94,7 +94,8 @@ Validators compute the [safe target](./lmd_ghost.md#safe-target-selection) they' deciding which finality vote to cast on the next slot. The safe target is computed based on the votes received in the current slot. -It is LMD-GHOST run with a two-thirds weight threshold instead of a plain majority, so it +It is LMD-GHOST again, but run over just the votes that arrived this slot and with a +two-thirds weight threshold, where head selection applies none. The safe target therefore sits at or behind the head and advances only once a branch is backed by a supermajority. Deriving targets from it is what stops [3SF-mini](./3sf_mini.md) from justifying a branch the network has not visibly converged on. @@ -105,6 +106,8 @@ Validators merge the aggregated attestations they have in their "new attestation into their fork-choice view, and recompute their head. This is the slot's second and last promotion point; the first is the proposer's, just -before it builds. Between promotions a vote sits in the buffer without weight, which is -what keeps a validator's fork-choice view from shifting under it mid-slot. See -[why staged promotion](./lmd_ghost.md#why-staged-promotion) for the reasoning. +before it builds. Until a vote is promoted it carries no weight in head selection, which is +what keeps a validator's fork-choice view from shifting under it mid-slot. The safe target +is the exception: it reads the unpromoted buffer directly, which is how it stays a view of +this slot alone. See [why staged promotion](./lmd_ghost.md#why-staged-promotion) for the +reasoning. From ace87b2fd4c7194527f976470a442382e1815ec1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1s=20Gr=C3=BCner?= <47506558+MegaRedHand@users.noreply.github.com> Date: Wed, 12 Aug 2026 16:01:28 -0300 Subject: [PATCH 5/7] docs: add architecture doc --- docs/architecture.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 docs/architecture.md diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..5e3cceb2 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,27 @@ +# Architecture + +The current ethlambda architecture consists of two genservers: one that manages the libp2p swarm (`P2PServer`), which runs in its own thread; and another that manages consensus events (`BlockChainServer`). Both genservers message each other, sending blocks for processing or publishing new blocks, for example. Independent of this, genservers have a reference to the underlying storage engine (the `Store`). Another process, the axum web server, is responsible for exposing data extracted from the running node to the outside world. + +> Note: for what a genserver is, read [this blogpost on the `spawned` crate](https://blog.lambdaclass.com/introducing-spawned-erlang-style-actors-for-rust/). + +## The `BlockChainServer` + +This genserver is responsible for serializing consensus updates and processing consensus events. It uses self-messages on a timer to drive the slot clock, and it receives messages from the `P2PServer` when new blocks or attestations are received. + +When each slot interval is reached, the `BlockChainServer` performs any validator duties due in that interval. For example, during the vote propagation interval, it gossips its votes to the network; during the vote aggregation interval, if an aggregator, it aggregates votes received from the network; and so on. + +This genserver has an aggregation worker, that concurrently performs vote aggregation. Once enough signatures are received in the vote propagation interval, the aggregation worker is started with a snapshot of the votes received so far. The snapshot includes votes for the current slot and also previous aggregated payloads and signatures for further aggregation. These are selected according to perceived usefulness. Once the aggregation worker finishes, it sends the aggregated payload back to the `BlockChainServer`, which gossips it to the network. + +## The `P2PServer` + +This genserver is responsible for managing the libp2p swarm, receiving events from it, and sending messages to it. It receives messages from the `BlockChainServer` to gossip new blocks or attestations, forwarding those to the swarm. When new blocks or attestations are received from the network, it sends them to the `BlockChainServer`. The initial bootstrapping of the swarm is done by connecting to a set of bootstrap nodes given by the user. + +Block requests sent by the `BlockChainServer` are handled by `P2PServer` too, which forwards them to the swarm. Any failed requests are retried a number of times before giving up. + +## `Store` + +The `Store` follows a similar approach as ethrex's. It is a safe and easy to use interface, that uses a pluggable key-value store (`StorageBackend`) as the underlying storage. The `Store` is used by the whole node to access heavy data, with the `BlockChainServer` being the only writer of consensus state. + +## HTTP API + +We use `axum` as our API router. API requests are served in tokio tasks. We use the same router to serve metrics, but these can be configured to be served in different ports. From debf7d0f1affc23730f610e7ec29d05d3318196e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1s=20Gr=C3=BCner?= <47506558+MegaRedHand@users.noreply.github.com> Date: Thu, 13 Aug 2026 17:23:48 -0300 Subject: [PATCH 6/7] docs: expand the architecture doc The page described the two actors in four paragraphs and was never listed in SUMMARY.md, so mdbook did not render it at all. Add a component diagram, the actor protocol table, and sections covering the tick loop, block import, off-loop aggregation, the sync gate, pending-parent backfill, chain events, the P2P swarm split, the Store and the startup sequence. Every claim was checked against the code it describes. Link the page from SUMMARY.md under a new Design part, and mention it in the introduction. --- docs/SUMMARY.md | 4 + docs/architecture.md | 206 +++++++++++++++++++++++++++++++++++++++++-- docs/introduction.md | 5 +- 3 files changed, 205 insertions(+), 10 deletions(-) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index a7ac16fa..d5f4234d 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -2,6 +2,10 @@ [Introduction](./introduction.md) +# Design + +- [Architecture](./architecture.md) + # Consensus - [Slots and Intervals](./slots_and_intervals.md) diff --git a/docs/architecture.md b/docs/architecture.md index 5e3cceb2..e15802d4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,27 +1,215 @@ # Architecture -The current ethlambda architecture consists of two genservers: one that manages the libp2p swarm (`P2PServer`), which runs in its own thread; and another that manages consensus events (`BlockChainServer`). Both genservers message each other, sending blocks for processing or publishing new blocks, for example. Independent of this, genservers have a reference to the underlying storage engine (the `Store`). Another process, the axum web server, is responsible for exposing data extracted from the running node to the outside world. +A running node is two actors, a few helper tasks, and one shared `Store`. The actors are +genservers: each owns its state, and the only way in is a message. + +- **`BlockChainServer`** (`crates/blockchain/src/lib.rs`) drives consensus. It runs the slot + clock, performs validator duties, imports blocks and attestations, and is the sole writer + of consensus state. +- **`P2PServer`** (`crates/net/p2p/src/lib.rs`) owns the network side: gossip publication, + request/response, peer bookkeeping and long-range sync. + +Everything else hangs off those two: an aggregation worker on a blocking thread, the libp2p +swarm loop on its own task, and the axum servers that expose the node to the outside world. > Note: for what a genserver is, read [this blogpost on the `spawned` crate](https://blog.lambdaclass.com/introducing-spawned-erlang-style-actors-for-rust/). +```text + Tick (self-message, one per interval) + ┌──────────────────────────────┐ + ▼ │ + ┌─────────────────────────────┐ │ ┌────────────────────┐ + │ BlockChainServer │───────────────┘ │ aggregation │ + ┌───▶│ actor, sole state writer │─────── jobs ────────▶│ worker │ + │ │ │◀───── aggregates ────│ (blocking thread) │ + │ └──────────────┬──────────────┘ └────────────────────┘ + │ │ + │ P2PToBlockChain │ BlockChainToP2P + │ new block/vote │ publish, fetch block + │ ▼ + │ ┌─────────────────────────────┐ SwarmCommand ┌────────────────────┐ + └────│ P2PServer │────────────────▶│ swarm adapter │ + │ actor, gossip + req/resp │◀────────────────│ (libp2p task) │◀══▶ peers + └─────────────────────────────┘ SwarmEvent └────────────────────┘ + + ┌────────────────────────────────────────────────────────────────────┐ + │ Store: pluggable key-value backend + in-memory fork-choice buffers │ + │ written by BlockChainServer, read by P2PServer and the API servers │ + └────────────────────────────────────────────────────────────────────┘ +``` + +## Actor protocols + +The two actors never share memory: they talk over the typed protocols in +`crates/net/api/src/lib.rs`. + +| Direction | Messages | +| --- | --- | +| `BlockChain` → `P2P` | `publish_block`, `publish_attestation`, `publish_aggregated_attestation`, `fetch_block` | +| `P2P` → `BlockChain` | `new_block` (tagged `Gossip` or `Sync`), `new_attestation`, `new_aggregated_attestation` | + +Both refs start as `None`. The `InitP2P` and `InitBlockChain` messages fill them in right +after spawn, so neither actor needs the other to exist at construction time. + ## The `BlockChainServer` -This genserver is responsible for serializing consensus updates and processing consensus events. It uses self-messages on a timer to drive the slot clock, and it receives messages from the `P2PServer` when new blocks or attestations are received. +### The tick loop + +The actor schedules its first `Tick` for genesis time, and every handler re-arms the next one +at the following interval boundary. A handler that overruns that boundary re-arms with a zero +delay instead, so the interval it just missed still gets its duty. The handler derives +`(slot, interval)` from the wall clock, compares it against the store's own interval counter, +and skips ticks the store already passed. + +`store::on_tick` then walks the store clock forward one interval at a time, fast-forwarding +if it fell more than a slot behind, so a late tick still runs each interval's bookkeeping in +order. That walk and the actor split the duties: + +| Interval | In `store::on_tick` | In the actor | +| --- | --- | --- | +| 0 | accept new attestations, if we propose this slot | nothing: the build ran at the previous interval 4 | +| 1 | nothing | produce attestations, arm the early-aggregation check | +| 2 | nothing | start the aggregation session | +| 3 | update the safe target | nothing | +| 4 | accept accumulated attestations | build and publish the next slot's block | + +See [Slots and Intervals](./slots_and_intervals.md) for what each duty means at the protocol +level, and why the proposer builds one interval early. + +The actor also advances its XMSS signing keys on every tick, and catches them up to the +current slot once at spawn. The keys are one-time and slot-bound, so a node that skipped this +would have nothing left to sign with. + +### Block import + +Blocks take the same path whether they arrived on gossip or came back from a `BlocksByRange` +sync request. The actor verifies the signature, then runs the state transition +(`crates/blockchain/state_transition`): `process_slots` advances the pre-state through empty +slots, `process_block` validates the header and applies the block's attestations, and the +result is rejected unless the recomputed state root matches the one the proposer committed +to. Justification and finalization move as part of that transition, following the +[3SF-mini](./3sf_mini.md) rules. The actor then writes the block and its post-state, and +recomputes the head with [LMD GHOST](./lmd_ghost.md) (`crates/blockchain/fork_choice`). + +### Aggregation off the message loop -When each slot interval is reached, the `BlockChainServer` performs any validator duties due in that interval. For example, during the vote propagation interval, it gossips its votes to the network; during the vote aggregation interval, if an aggregator, it aggregates votes received from the network; and so on. +XMSS proving costs hundreds of milliseconds, so it cannot run on the actor loop: a blocked +actor stops importing blocks. The actor instead snapshots aggregation inputs from the store, +ranks candidates by consensus value, and hands a job list to a `spawn_blocking` worker +(`crates/blockchain/src/aggregation.rs`). The worker holds no store access, streams one +`AggregateProduced` message back per finished job, and ends with `AggregationDone`. The actor +publishes each result on gossip when it arrives. -This genserver has an aggregation worker, that concurrently performs vote aggregation. Once enough signatures are received in the vote propagation interval, the aggregation worker is started with a snapshot of the votes received so far. The snapshot includes votes for the current slot and also previous aggregated payloads and signatures for further aggregation. These are selected according to perceived usefulness. Once the aggregation worker finishes, it sends the aggregated payload back to the `BlockChainServer`, which gossips it to the network. +A soft deadline cancels the worker through a `CancellationToken`, so an overrunning slot +cannot eat the next one. The session can also start up to `EARLY_AGGREGATION_WINDOW` before +interval 2, once two thirds of the expected signatures are in. Either entry point counts as +the slot's one session, so a slot aggregates once. Starting early buys proving time, not an +earlier publication: the worker holds each finished aggregate until the interval-2 boundary +before delivering it to the actor. + +Block import runs a second, smaller aggregation path. `reaggregate.rs` splits an imported +block's merged proof back into per-attestation aggregates and folds them into the local pool, +which is how a node that only saw a vote inside a block gets its fork-choice weight. +Aggregators go one step further and republish those aggregates on gossip. Each split runs a +fresh SNARK, so `reaggregate.rs` caps how many it does per block, and the actor skips the +whole path while the node is catching up. + +### Sync gate + +`sync_status.rs` tracks how far the local head lags the slot clock. Past the threshold the +node reports itself syncing and stops attesting and proposing, since a head derived from a +partial view is not worth voting for. A hysteresis band stops the state from flapping at the +boundary, and a network-wide stall (nobody else is ahead either) leaves the node synced so +its validators can help the chain recover. The same status feeds the `lean_node_sync_status` +metric and, through a shared controller, the `/lean/v0/node/syncing` endpoint. +`--disable-duty-sync-gate` reduces the gate to observe-only. + +### Missing parents + +The actor cannot import a block whose parent is unknown, so it parks the block in +`pending_blocks` under its parent root and records the deepest missing ancestor it can find, +walking back through already-stored pending blocks, in `pending_block_parents`. That ancestor +is what it asks the `P2PServer` to fetch. Once the ancestor lands, the actor cascades down +the parent index and re-imports every block that was waiting. + +### Chain events + +The actor is the sole publisher on an `EventBus` (`crates/blockchain/src/events.rs`), which +carries seven topics: head moves, imports and gossip sightings of blocks, single votes and +aggregates, plus justification and finalization updates. The bus is best-effort: +emission never blocks the actor, and a slow subscriber loses events instead of +back-pressuring consensus. The API server subscribes one receiver per SSE client. ## The `P2PServer` -This genserver is responsible for managing the libp2p swarm, receiving events from it, and sending messages to it. It receives messages from the `BlockChainServer` to gossip new blocks or attestations, forwarding those to the swarm. When new blocks or attestations are received from the network, it sends them to the `BlockChainServer`. The initial bootstrapping of the swarm is done by connecting to a set of bootstrap nodes given by the user. +Only the swarm adapter task touches the `libp2p::Swarm`. The actor sends it `SwarmCommand`s +(publish, dial, send request, send response) and receives `SwarmEvent`s back as actor +messages. That split keeps non-`Clone` swarm types (response channels, for one) out of the +typed protocol, and keeps the adapter polling network I/O while the actor is busy with a +message. + +What the actor does with those events: -Block requests sent by the `BlockChainServer` are handled by `P2PServer` too, which forwards them to the swarm. Any failed requests are retried a number of times before giving up. +- **Gossip.** It decodes blocks, aggregates and per-subnet attestations, then forwards them + to the `BlockChainServer`. The node computes its subscriptions once at startup from its + validator set and aggregator role, and never revisits them. +- **Status.** Sent on the first connection to a peer, not on every redundant one. When a peer + reports a head ahead of ours, the actor opens a long-range sync range or extends the one it + has, then requests `BlocksByRange` batches one at a time, dropping peers that fall behind + the range. +- **`BlocksByRoot`.** Backs the `fetch_block` requests above. Retries use exponential backoff + and prefer a peer that has not already failed for that root, falling back to the full + connected set once every peer has failed. -## `Store` +The `P2PServer` holds its own `Store` clone, which it only ever reads, so it answers `Status` +and `BlocksBy*` requests without a round trip through the consensus actor. -The `Store` follows a similar approach as ethrex's. It is a safe and easy to use interface, that uses a pluggable key-value store (`StorageBackend`) as the underlying storage. The `Store` is used by the whole node to access heavy data, with the `BlockChainServer` being the only writer of consensus state. +## The `Store` + +The `Store` follows a similar approach as ethrex's: a safe, easy to use interface over a +pluggable key-value backend (`StorageBackend`, RocksDB for a normal node and in-memory for +tests and the Hive test driver). +Cloning one shares the backend, the state LRU cache, and the in-memory buffers that fork +choice runs on: the new and known attestation payload buffers, the latest votes, and the +gossip signatures awaiting aggregation. The node never persists those buffers, since they +only matter for the slot they belong to. + +Every component gets a clone, but only the `BlockChainServer` writes consensus state. A +reader can therefore hold a handle without interleaving with a state transition. + +See [Data Storage](./data_storage.md) for the table layout, the snapshot/diff scheme used for +states, and pruning rules. ## HTTP API -We use `axum` as our API router. API requests are served in tokio tasks. We use the same router to serve metrics, but these can be configured to be served in different ports. +We use `axum` as our API router, with requests served in tokio tasks. Handlers read node state +without messaging the actors: the `Store` is router state, and the `EventBus` and the runtime +controllers arrive as extension layers. Metrics and debug endpoints live in their own routers, +on a port you configure separately: distinct ports bind two independent servers, equal ports +merge all three routers onto a single listener. See [HTTP API](./rpc.md) for the endpoint +reference. + +## Startup + +`bin/ethlambda/src/main.rs` wires the node in a fixed order. Everything before the actors +spawn is fail-fast, so a misconfigured node stops at boot instead of hours later. + +1. Install the tracing subscriber, parse CLI options, register metrics, raise the + file-descriptor limit for RocksDB's unbounded table cache. +2. Load the node key, then genesis config, validator config, bootnode ENRs and validator keys. +3. Open the database, then pick an anchor: resume from disk if the on-disk head is recent + enough, otherwise [checkpoint sync](./checkpoint_sync.md) from the configured URLs, and + otherwise build the genesis state. A stale database with no checkpoint URL configured is + resumed anyway, with a warning, since that is the setup the node was given. +4. Build the shared handles: aggregator controller, sync-status controller, event bus, and + the subnet set that both the swarm and the actor need to agree on. +5. Spawn `BlockChainServer` (which schedules its first tick for genesis time), build the + swarm, spawn `P2PServer`, and wire the two together with `InitP2P` and `InitBlockChain`. +6. Spawn the API and metrics servers. + +Shutdown runs in reverse: the first ctrl+c stops both actors and cancels the servers' +shutdown token, and three more force the process to exit if a graceful stop hangs. + +> Note: booting with `HIVE_LEAN_TEST_DRIVER=1` short-circuits everything from step 2 on and +> exposes only the Hive test-driver endpoints, so a driver run never touches the node key, +> the genesis config or any other consensus prerequisite. diff --git a/docs/introduction.md b/docs/introduction.md index 4ffe4caf..4c406ab4 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -4,8 +4,11 @@ consensus client, written in Rust. This book collects the design notes and operator-facing references for ethlambda. -It is split into two parts: +It is split into three parts: +- **Design** describes the shape of a running node: the + [architecture](./architecture.md) of its actors, workers and shared storage, + and how they are wired together at startup. - **Consensus** explains how the chain advances: the [slot and interval structure](./slots_and_intervals.md) that schedules every validator duty, the [3SF-mini](./3sf_mini.md) justification and finalization From b4a92f1a34505ae94e283d44a717d41ff22a5430 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1s=20Gr=C3=BCner?= <47506558+MegaRedHand@users.noreply.github.com> Date: Thu, 13 Aug 2026 17:24:26 -0300 Subject: [PATCH 7/7] docs: list the Development part in the introduction The introduction enumerated the book's parts but stopped at Operations, so Spec Deviations was reachable from the sidebar and invisible from the intro. --- docs/introduction.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/introduction.md b/docs/introduction.md index 4c406ab4..b056ae6d 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -4,7 +4,7 @@ consensus client, written in Rust. This book collects the design notes and operator-facing references for ethlambda. -It is split into three parts: +It is split into four parts: - **Design** describes the shape of a running node: the [architecture](./architecture.md) of its actors, workers and shared storage, @@ -19,6 +19,8 @@ It is split into three parts: [Prometheus metrics](./metrics.md), [checkpoint sync](./checkpoint_sync.md), and the [fork choice visualization](./fork_choice_visualization.md) served by the API. +- **Development** collects notes for contributors, starting with the + [spec deviations](./spec_deviations.md) where ethlambda departs from leanSpec. For build and contribution instructions, see the [`README`](https://github.com/lambdaclass/ethlambda/blob/main/README.md) and