Skip to content

test(core): add reranker real-model smoke and quality harness - #1233

Merged
phernandez merged 3 commits into
mainfrom
test/1231-reranker-quality-harness
Aug 11, 2026
Merged

test(core): add reranker real-model smoke and quality harness#1233
phernandez merged 3 commits into
mainfrom
test/1231-reranker-quality-harness

Conversation

@phernandez

Copy link
Copy Markdown
Member

Phases 2-3 of #1231 — the remaining automated items. Test-only: six files under
test-int/semantic/, zero production changes.

What's added

  • Real-model smoke (test_real_fastembed_reranker.py, semantic-marked): loads the
    actual jinaai/jina-reranker-v1-tiny-en cross-encoder through the production
    provider/factory path (no stubs), asserts the relevant doc wins with calibrated [0,1]
    scores and deterministic ordering, plus one end-to-end hybrid search through an indexed
    repository with reranking enabled.

  • Quality regression harness (test_reranker_quality.py + ranking corpus/metrics
    extensions): golden ranking-sensitive queries measured reranker-off vs reranker-on with
    real embeddings + real cross-encoder. Asserts on >= off for hit@1/MRR plus an absolute
    hit@5 floor so a silent rerank no-op cannot pass. Measured on the curated set:

    Mode hit@1 hit@5 MRR
    Reranking off 0.833 1.000 0.917
    Reranking on 1.000 1.000 1.000

    No queries needed exclusion; the real tiny model degraded nothing.

  • Latency benchmark (test_reranker_latency.py, semantic+benchmark-marked, report-
    focused with a generous catastrophic-regression ceiling only). Warmed local numbers,
    20 iterations:

    Search P50 P95
    Vector off 9.5 ms 10.1 ms
    Vector on 102.5 ms 238.4 ms
    Hybrid off 13.2 ms 17.6 ms
    Hybrid on 98.2 ms 463.8 ms

    ~90 ms P50 overhead for the local tiny cross-encoder — comfortably inside the latency
    headroom that motivated Add a rerank stage to search: ~half of LoCoMo benchmark misses are ranking failures, and there is ~20x latency headroom #950.

Verification

  • The three new test files: pass, run twice for determinism (16s total once models are
    cached).
  • just typecheck, just lint: clean.
  • Note: test_semantic_quality[asyncio-postgres-fastembed] can exceed the 120s per-test
    timeout on a loaded local machine — reproduced identically on main, pre-existing and
    unrelated to this PR.

Ticks the Phase 2 fastembed-smoke and both Phase 3 boxes on #1231. Remaining there: the
deferred live LiteLLM run and the optional live smoke. These numbers also set the baseline
for revisiting #951 (entity boosting) and closing #618.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CPdSXDbYyhyZ1TwgFnpEv8

Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant