feat(kv-store): expose read-only lmdb transactions - #25280
Open
spalladino wants to merge 5 commits into
Open
Conversation
Adds START_READ_TX / CLOSE_READ_TX messages and an optional txId on GET and START_CURSOR, so the JS side can hold one LMDB read transaction open across many reads and iterations and see a single consistent snapshot. LMDBStore gets a get() overload that reads against a caller-supplied read transaction. LMDBStoreWrapper keeps a registry of read transactions mirroring the cursor registry; because a read transaction must never be used by two threads at once, each one carries a mutex that every get and every cursor bound to it locks. Cursors record that mutex so advance_cursor serializes against sibling cursors and gets on the same snapshot.
`store.readOnlyTransaction(cb)` opens a real LMDB read transaction and keeps it open for the whole callback, so every read inside sees one snapshot. Readers do not go through the writer queue, so a write can commit while the callback runs without the callback observing it. The transaction is propagated through an AsyncLocalStorage, so container reads (`map.getAsync`, `entriesAsync`, ...) inside the callback hit the snapshot without being handed the transaction explicitly. Nested calls reuse the enclosing transaction, matching how `transactionAsync` handles recursion. Each open snapshot consumes an LMDB reader slot, so it acquires from the same semaphore as cursors; cursors bound to a snapshot skip acquisition since they reuse its slot, which the store now tracks per cursor id to avoid over-releasing on close. `readOnlyTransaction` is added to `AztecAsyncKVStore`; the other backends have no snapshot of their own that outlives an operation, so they delegate to their regular transaction, which gives the callback a consistent view.
The aztec-up bridge_and_claim smoke test intermittently fails with BBApiException: Failed to verify the generated proof! during ClientIVC private-kernel proving. This is unrelated to the kv-store/lmdb changes in this branch and matches the existing pattern of proving-related flakes already catalogued for this and adjacent aztec-up tests.
… corrupted artifact
The failure was a corrupted one-off bb build artifact replayed from cache, not a recurring flake; suppressing this signature would mask real prover regressions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
AztecLMDBStoreV2only exposestransactionAsync, which opens a WRITE transaction serialized through the writer queue. The TSReadTransactiongives no snapshot consistency: everyGETmessage opens and aborts a throwaway LMDB read tx in C++, so two consecutive reads can straddle a commit. LMDB supports many concurrent readers that never block the writer, but that capability was not exposed to TS.Approach
START_READ_TX/CLOSE_READ_TXmessages to the node addon protocol, plus an optionaltxIdonGETandSTART_CURSOR. The C++ wrapper keeps a registry of live read transactions (mirroring the existing cursor registry), each guarded by a mutex since an LMDB read tx must not be used concurrently across the libuv pool threads; cursors opened against a shared tx serialize on that same mutex.lmdblib::LMDBStore::getgains an overload that reads through a caller-provided read transaction instead of opening its own.store.readOnlyTransaction(callback)opens one C++ read tx and propagates it viaAsyncLocalStorage, so plain container reads (map.getAsync, iteration) inside the callback hit the snapshot automatically. Nested calls reuse the ambient read or write tx, matchingtransactionAsyncsemantics.maxReaders - 1semaphore; cursors created inside a snapshot skip slot acquisition since they share the tx's reader slot.API changes
AztecAsyncKVStoregainsreadOnlyTransaction<T>(callback: () => Promise<T>): Promise<T>: runs the callback against a consistent read-only snapshot without blocking concurrent writers. The sqlite-opfs, indexeddb, and v1 lmdb implementations delegate to their existing transaction machinery (consistent view, no snapshot concurrency).