Skip to content

fix(archiver): batch multi-key point reads into one LMDB round trip - #25282

Open
spalladino wants to merge 1 commit into
merge-train/spartan-v5from
spl/batched-kv-gets
Open

fix(archiver): batch multi-key point reads into one LMDB round trip#25282
spalladino wants to merge 1 commit into
merge-train/spartan-v5from
spl/batched-kv-gets

Conversation

@spalladino

Copy link
Copy Markdown
Contributor

Context

Every getAsync on the lmdb-v2 kv-store costs a full JS→C++ msgpack round trip, and the archiver has hot paths doing one point read per tx effect or per log: getBlock awaited a sequential read per tx (120-tx block = 120 serialized round trips), and getNoteHashesAndNullifiers (the includeEffects path of getLogsByTags) issued one message per tx hash. The wire protocol's GET request already accepts an array of keys and the native handler returns one slot per key in order — nothing on the TS side used it.

Approach

  • ReadTransaction.getMany(keys) sends one GET message for all keys; WriteTransaction overrides it, resolving each key against the pending batch first (pending writes and removes behave exactly like N sequential get calls) and batch-fetching only the rest. The single-key get now shares the same pending-lookup helper.
  • AztecAsyncMap.getManyAsync(keys) is added to the interface; lmdb-v2 maps use the batched read, other backends (lmdb-v1, sqlite-opfs, indexeddb, lmdb-v2 multimap) use a Promise.all of getAsync.
  • Converted the archiver's N-point-read loops: getBlock's tx-effect loop, getNoteHashesAndNullifiers, and the per-block log reads behind getPrivateLogsForBlock/getPublicLogsForBlock. Warn messages, missing-key behavior, and output order are unchanged.

Measured (interleaved A/B, 3 runs each): getBlock on a 120-tx block 80–84ms → 64–71ms (~17% faster; most of the win is not interleaving deserialization between sequential awaits). getNoteHashesAndNullifiers with 100 hashes was already pipelined via Promise.all, so it drops 100 messages to 1 with a small wall-clock gain (~7%).

API changes

AztecAsyncMap gains getManyAsync(keys: K[]): Promise<(V | undefined)[]>; any external implementations of the interface need the method (a Promise.all over getAsync is a valid implementation).

…d trip

The lmdb-v2 wire protocol always supported multi-key GET, but every
getAsync sent a single-key message. Add getMany to the read and write
transactions (overlaying pending batch state) and getManyAsync to
AztecAsyncMap, and use it where the archiver did one point read per tx
effect or per log: getBlock, getNoteHashesAndNullifiers, and
per-block log reads.
@github-actions github-actions Bot added the port-to-next Forward-port this merged PR into next label Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

port-to-next Forward-port this merged PR into next

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant