Skip to content

fix(archiver): speed up log-by-tag queries - #25254

Open
spalladino wants to merge 4 commits into
merge-train/spartan-v5from
spl/faster-get-private-logs-by-tags
Open

fix(archiver): speed up log-by-tag queries#25254
spalladino wants to merge 4 commits into
merge-train/spartan-v5from
spl/faster-get-private-logs-by-tags

Conversation

@spalladino

@spalladino spalladino commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Context

getPrivateLogsByTags was reported at 100-200ms per call in production, ~10x slower than other node RPC calls. The existing node_rpc_perf bench couldn't see it (~0.2ms) because it measured a single random tag; measured PXE traffic at the node boundary is very different: ~100 tags per call (median 84, p95 100), includeEffects on, no fromBlock, and a near-total miss rate (0.45% of tags match; 86% of calls return nothing). Profiling with representative benches attributed the cost to: per-tag scans running sequentially, each paying 2-4 native msgpack cursor round trips (10-entry pages vs a 20-log limit); queries queueing behind writes on lmdb-v2's single writer queue; and, for log-heavy responses, re-hydrating thousands of Fr fields through layered Zod schemas and a Buffer-to-hex-to-BigInt round trip.

Approach

Queries stay transactional: they still run in db.transactionAsync with exactly the same snapshot-consistency semantics as before (and therefore still serialize with writes — an earlier revision of this PR moved them off the writer queue and was deliberately rolled back to keep the consistency guarantee; making writes faster is the follow-up lever for contended latency).

Within that unchanged envelope:

  • Archiver log store: per-tag scans run 8-wide via asyncPool over the transaction's snapshot instead of one at a time, preserving per-tag result order.
  • kv-store lmdb-v2: range scans with limit <= 128 are fetched as a single one-page cursor request (one native round trip, no cursor slot held). A narrow fast path in WriteTransaction.iterate delegates to that one-page read when the pending batch is empty — safe because nothing pending can shadow or suppress a committed entry — so it also applies to reads inside a transaction.
  • Foundation: hexSchemaFor collapses its refine/refine/transform chain into a single transform, and fromHexString builds field elements via BigInt('0x…') directly instead of round-tripping through a Buffer (~5x faster per field, benefits all hex deserialization).

Measured impact (uncontended, vs base):

  • Production-shaped call (100 tags, ~all misses, includeEffects), in-process: 5.3ms → 3.2ms (~40%).
  • 100-tag all-hits call: in-process 7.8ms → ~5.5ms; over HTTP 19.2ms → ~15ms.
  • Queued behind 5 concurrent 100-tag queries: 41ms → ~33ms (reads still serialize; each item in the queue got cheaper).
  • Client-side parse of a 100-log response: 2.24ms → 1.9ms.

Benches and tests added:

  • node_rpc_perf bench now mints to private so blocks contain real private logs, and adds getPrivateLogsByTags_100tags shaped after the measured recipient-sync traffic (100 tags, 1 hit + 99 misses, includeEffects, referenceBlock; ~3.4ms avg) plus getPrivateLogsByTags_100tags_allhits and getPublicLogsByTags_100tags stress variants, alongside the existing single-tag series.
  • New node_rpc_private_logs_perf bench: in-process vs HTTP matrix with real vs random tags, control calls, and a queueing contention probe.
  • New log_store_write_contention test: asserts a tag query racing queued write transactions returns results identical to an uncontended query.

@github-actions github-actions Bot added the port-to-next Forward-port this merged PR into next label Aug 18, 2026
Queries that omit referenceBlock now pin themselves to the current chain tip: the tip caps every per-tag scan and is re-checked after the scans, and a query whose anchor is unwound mid-flight is re-run against the new tip (3 attempts) instead of returning a torn read. Explicit-referenceBlock queries keep throwing the reorg error as before.
Tag log queries run inside db.transactionAsync again, so they serialize with queued writes by design and cannot
return a torn result; the implicit self-anchoring and post-scan anchor re-check are dropped. The optimizations that
do not change consistency semantics are kept: 8-wide parallel per-tag scans, and one-page cursor reads for bounded
scans (now also reached from inside a write transaction whose batches are empty).
@spalladino spalladino changed the title fix(archiver): unblock and speed up log-by-tag queries fix(archiver): speed up log-by-tag queries Aug 20, 2026
@AztecBot

Copy link
Copy Markdown
Collaborator

Flakey Tests

🤖 says: This CI run detected 1 tests that failed, but were tolerated due to a .test_patterns.yml entry.

\033FLAKED\033 (8;;http://ci.aztec-labs.com/0c4e03ece8955af1�0c4e03ece8955af18;;�):  yarn-project/end-to-end/scripts/run_test.sh simple src/multi-node/slashing/slash_veto_demo.test.ts (246s) (code: 0) group:e2e-p2p-epoch-flakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

port-to-next Forward-port this merged PR into next

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants