fix(archiver): batch multi-key point reads into one LMDB round trip - #25282
Open
spalladino wants to merge 1 commit into
Open
fix(archiver): batch multi-key point reads into one LMDB round trip#25282spalladino wants to merge 1 commit into
spalladino wants to merge 1 commit into
Conversation
…d trip The lmdb-v2 wire protocol always supported multi-key GET, but every getAsync sent a single-key message. Add getMany to the read and write transactions (overlaying pending batch state) and getManyAsync to AztecAsyncMap, and use it where the archiver did one point read per tx effect or per log: getBlock, getNoteHashesAndNullifiers, and per-block log reads.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Every
getAsyncon the lmdb-v2 kv-store costs a full JS→C++ msgpack round trip, and the archiver has hot paths doing one point read per tx effect or per log:getBlockawaited a sequential read per tx (120-tx block = 120 serialized round trips), andgetNoteHashesAndNullifiers(theincludeEffectspath ofgetLogsByTags) issued one message per tx hash. The wire protocol'sGETrequest already accepts an array of keys and the native handler returns one slot per key in order — nothing on the TS side used it.Approach
ReadTransaction.getMany(keys)sends oneGETmessage for all keys;WriteTransactionoverrides it, resolving each key against the pending batch first (pending writes and removes behave exactly like N sequentialgetcalls) and batch-fetching only the rest. The single-keygetnow shares the same pending-lookup helper.AztecAsyncMap.getManyAsync(keys)is added to the interface; lmdb-v2 maps use the batched read, other backends (lmdb-v1, sqlite-opfs, indexeddb, lmdb-v2 multimap) use aPromise.allofgetAsync.getBlock's tx-effect loop,getNoteHashesAndNullifiers, and the per-block log reads behindgetPrivateLogsForBlock/getPublicLogsForBlock. Warn messages, missing-key behavior, and output order are unchanged.Measured (interleaved A/B, 3 runs each):
getBlockon a 120-tx block 80–84ms → 64–71ms (~17% faster; most of the win is not interleaving deserialization between sequential awaits).getNoteHashesAndNullifierswith 100 hashes was already pipelined viaPromise.all, so it drops 100 messages to 1 with a small wall-clock gain (~7%).API changes
AztecAsyncMapgainsgetManyAsync(keys: K[]): Promise<(V | undefined)[]>; any external implementations of the interface need the method (aPromise.allovergetAsyncis a valid implementation).