fix(rag): tokenize CJK text so BM25 retrieval can score non-Latin queries

[kuroudo-ai]

Problem

retrieve_parsed_document silently returns meaningless results for Japanese and Chinese queries. Every chunk scores 0.0, but top_k hits are still returned, so unrelated chunks come back looking like search results. Nothing raises.

On master:

hits = retrieve_parsed_document(chunks, "国際標準化", top_k=3)
# 3 hits, every score 0.0, first hit unrelated

This is the failure mode that matters most for a RAG pipeline: the retriever reports success, and an unrelated chunk is handed to the model as evidence.

Cause

BM25Retriever.from_defaults defaults to token_pattern=r"(?u)\b\w\w+\b", which assumes whitespace-delimited words. Japanese and Chinese are not delimited, so a phrase such as 国際標準化に関する動向 becomes one token and the query token 国際標準化 never equals a document token.

Fix

Pass a token pattern that splits CJK characters individually and leaves every other script on the existing two-or-more-character rule:

(?u)[^\W<CJK ranges>]{2,}|[<CJK ranges>]

Because the alternation excludes CJK from the first branch only, Latin, Cyrillic and other alphabets tokenize exactly as before.

skip_stemming=True was considered and deliberately not used: it degrades English retrieval while adding no improvement for CJK.

Verification

• Added ScriptAwareRetrievalTests covering Japanese, Chinese and Latin. Removing the fix fails the Japanese and Chinese cases while the Latin case keeps passing — the tests fail for the reason they claim to.

• bash scripts/verify.sh passes: 429 passed, total coverage 84.78%.

• Measured retrieval quality on a mixed-language corpus:

  query	before	after
  国際標準化	0.0000, unrelated chunk ranked first	2.2514, correct chunk ranked first
  人材育成	0.0000, unrelated chunk ranked first	1.7285, correct chunk ranked first

• Latin-only corpus, four queries compared before/after: identical scores and identical ranking.

One caveat worth stating explicitly: on a mixed-language corpus, English scores do move (for example 1.056 → 2.068). That is not a regression in tokenization — CJK chunks now yield more tokens, which changes average document length and IDF for the whole corpus, and BM25 scores are relative to those statistics. Ranking is unaffected, and the Latin-only comparison above isolates the tokenizer itself.