Skip to content

docs: publish translated docs in twelve languages and the tool that maintains them - #3280

Open
maxisbey wants to merge 4 commits into
mainfrom
docs/translations
Open

docs: publish translated docs in twelve languages and the tool that maintains them#3280
maxisbey wants to merge 4 commits into
mainfrom
docs/translations

Conversation

@maxisbey

@maxisbey maxisbey commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Ships the docs in twelve languages — de, es, fr, hi, ja, ko, pt (Brazilian), ru,
tr, uk, zh and zh-hant, each at /<code>/ — machine-translated from the English
pages, plus the small tool that keeps them current. English stays
the only source; everything under i18n/<lang>/pages/ is generated. Supersedes
#3231 and #3265 (same goal; this takes the review of both into account —
notably no anchor edits to the English docs, no nav-label subsystem, no
second-model gate, no state file).

  • scripts/docs/translations.py — three commands: status, translate --lang X [--pages …], stage --lang X. Pages are split at ## sections; only sections whose English changed are re-translated
    and everything else is carried forward byte-for-byte, so a one-line English
    edit is a one-line translated diff. Heading ids come from the site renderer
    itself and are pinned into the translation; code blocks are re-imposed from the
    English mechanically and link targets are checked against it. Provenance (section hashes) lives
    in each generated page's front matter — no side state.
  • Corrections are inputs: i18n/<lang>/instructions.md (register, voice,
    typography) and glossary.json (terms, keep-list, banned renderings), then
    translate --pages … for the affected pages; the generated pages are never
    hand-edited.
  • Build: language sites are staged (English overlaid with translations, a
    short notice after each H1 — machine-translated / behind the English page /
    shown in English), sidebar titles come from each translated page's own H1,
    and language builds are non-strict, so an English-only PR can't fail because
    of translations; a translation that no longer fits its English page is served
    in English until refreshed. The language sites add about a minute to the docs
    build in total; the API reference is linked, not rebuilt per language.
  • The language switcher shows code - name and keeps you on the current page
    when you change language. docs/translations.md explains all this to readers.

Intentional behavior changes

None to the SDK. The docs build now also produces the twelve language sites;
anthropic is added in a non-default translate dependency group (normal
installs don't get it).

Review notes

  • Registers/terminology are provisional pending native review — Portuguese has a
    reviewer lined up; reading order: i18n/general-prompt.md
    i18n/pt/instructions.md + glossary.json → a few pages under
    i18n/pt/pages/docs/translations.md → the tool.
  • Known follow-ups: CJK search segmentation needs a Zensical bump (blocked on
    packaging, tracked separately); French spacing uses ordinary rather than
    no-break spaces (the model won't emit U+00A0; a small deterministic pass could); the "API Reference" nav label stays English
    on language sites; a handful of terminology consistency notes for native
    reviewers are listed in i18n/<lang>/glossary.json notes as provisional.
  • Running the tool needs ANTHROPIC_API_KEY; nothing in CI calls the API.

AI Disclaimer

…tains them

Adds machine-translated documentation sites for Simplified Chinese,
Japanese, Korean and Brazilian Portuguese under /zh-CN/, /ja/, /ko/ and
/pt-BR/, generated from the English pages, which stay the only source.

scripts/docs/translations.py has three commands: status, translate and
stage. Pages are split at their sections and only sections whose English
changed are re-translated; the rest is carried forward byte-for-byte.
Heading ids come from the site renderer and are pinned into the
translation, code blocks are re-imposed from the English, and each
generated page records the section hashes it reflects in its own front
matter. Per-language inputs live in i18n/<lang>/instructions.md and
glossary.json; corrections go there, never into the generated pages.

The build stages each language (English overlaid with its translations
plus a short notice after each page title), takes sidebar titles from
the translated pages, links the single English API reference, and builds
language sites non-strictly so English-only changes never fail on
translations. A manually dispatched workflow refreshes a language and
opens a draft PR for its reviewers.
@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

📚 Documentation preview

Preview https://pr-3280.mcp-python-docs.pages.dev
Deployment https://568e2414.mcp-python-docs.pages.dev
Commit d37ff04
Triggered by @maxisbey
Updated 2026-08-11 14:15:14 UTC

Drops the dispatch workflow and the issue form, the translate flags that
served them (--grep, --limit, --dry-run), the reviewers field, the extra
exit-code and schema-validation plumbing, and the link-resolution
machinery in stage: a translation that no longer fits the current English
page is simply served in English with the notice. What remains is
translate, stage and status plus the build wiring.
Adds Deutsch, español, français, हिन्दी, русский, Türkçe, українська and
繁體中文 alongside the existing four, using the same language codes and
switcher labels ("de - Deutsch") as other Python documentation sites;
zh-CN and pt-BR become zh and pt (Brazilian Portuguese). Each new language
gets its instructions and glossary plus a full set of generated pages.

The language switcher now keeps you on the current page when you change
language instead of returning to the home page.
@maxisbey maxisbey changed the title docs: publish translated docs (zh-CN, ja, ko, pt-BR) and the tool that maintains them docs: publish translated docs in twelve languages and the tool that maintains them Aug 11, 2026
Comment thread scripts/docs/translations.py Fixed
@maxisbey
maxisbey marked this pull request as ready for review August 11, 2026 13:59

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 657 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="i18n/de/pages/advanced/middleware.md">

<violation number="1" location="i18n/de/pages/advanced/middleware.md:95">
P3: This sentence has an extra `)` which makes the warning text read incorrectly and can confuse readers in a critical deadlock explanation. It looks like a translation typo and can be fixed by removing the extra parenthesis.</violation>
</file>

<file name="i18n/languages.yml">

<violation number="1" location="i18n/languages.yml:32">
P2: Language switcher alternate metadata currently declares the Brazilian Portuguese site as generic `pt`, which can cause search engines and locale pickers to target the wrong Portuguese variant. Using a region-qualified hreflang here keeps `/pt/` aligned with its actual Brazilian content.</violation>
</file>

Note: This PR contains a large number of files. cubic only reviews up to 200 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.

Re-trigger cubic

Comment thread i18n/languages.yml
- code: pt # Brazilian Portuguese
name: português (Brasil)
theme: pt-BR
hreflang: pt

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Language switcher alternate metadata currently declares the Brazilian Portuguese site as generic pt, which can cause search engines and locale pickers to target the wrong Portuguese variant. Using a region-qualified hreflang here keeps /pt/ aligned with its actual Brazilian content.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At i18n/languages.yml, line 32:

<comment>Language switcher alternate metadata currently declares the Brazilian Portuguese site as generic `pt`, which can cause search engines and locale pickers to target the wrong Portuguese variant. Using a region-qualified hreflang here keeps `/pt/` aligned with its actual Brazilian content.</comment>

<file context>
@@ -0,0 +1,52 @@
+  - code: pt # Brazilian Portuguese
+    name: português (Brasil)
+    theme: pt-BR
+    hreflang: pt
+  - code: ru
+    name: русский язык
</file context>

Middleware-Kette zurückkehrt. Auf einen Request vom Server an den Client zu warten (`ctx.session.send_request(...)`,
eine Elicitation (Rückfrage bei der Person am Host)), während `initialize` verarbeitet wird, führt daher zu
einem **Deadlock der Verbindung**: Die Response, auf die du wartest, kann nie gelesen werden.
Fire-and-forget-Benachrichtigungen sind in Ordnung.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This sentence has an extra ) which makes the warning text read incorrectly and can confuse readers in a critical deadlock explanation. It looks like a translation typo and can be fixed by removing the extra parenthesis.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At i18n/de/pages/advanced/middleware.md, line 95:

<comment>This sentence has an extra `)` which makes the warning text read incorrectly and can confuse readers in a critical deadlock explanation. It looks like a translation typo and can be fixed by removing the extra parenthesis.</comment>

<file context>
@@ -0,0 +1,123 @@
+    Middleware-Kette zurückkehrt. Auf einen Request vom Server an den Client zu warten (`ctx.session.send_request(...)`,
+    eine Elicitation (Rückfrage bei der Person am Host)), während `initialize` verarbeitet wird, führt daher zu
+    einem **Deadlock der Verbindung**: Die Response, auf die du wartest, kann nie gelesen werden.
+    Fire-and-forget-Benachrichtigungen sind in Ordnung.
+
+## Die eine Middleware, die standardmäßig aktiv ist {#the-one-middleware-that-ships-on-by-default}
</file context>

Matches a heading's trailing attribute blocks with a pattern that reads
each block in one pass; the previous form could take a very long time on
a heading carrying many blocks followed by other text. Parsed results are
unchanged.

No-Verification-Needed: one regular expression rewritten with identical results, covered by a new unit test
# A heading's trailing attr_list block(s), matched with or without the whitespace
# attr_list itself needs, so blocks the model glued to CJK text or doubled are
# still seen; `body` is the last block's, the one attr_list reads.
HEADING_ATTRS = re.compile(r"(?:[ \t]*\{:?(?P<body>[^}\n]*)\})+[ \t]*$")
Comment thread i18n/README.md

```text
uv run --frozen python scripts/docs/translations.py status [--lang CODE]
uv run --frozen --group translate python scripts/docs/translations.py translate --lang CODE [--pages PATH ...]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 command_translate constructs the Anthropic client (translator = translator or anthropic_translator(), scripts/docs/translations.py:869) before the job loop, so a run whose jobs are all no-call reassembly jobs (English sections only reordered/removed) still exits 2 with ConfigError when the translate dependency group or ANTHROPIC_API_KEY is absent — even though it would make zero API requests. Constructing the client lazily, e.g. if any(job.open for job in jobs): translator = translator or anthropic_translator(), keeps the credential fail-fast for runs that do call the model while letting the offline reassembly path actually run offline.

Extended reasoning...

What the code does today. command_translate (scripts/docs/translations.py:861-883; this comment is anchored on i18n/README.md's tool section because the tracker's diff subset omits the script) runs translator = translator or anthropic_translator() unconditionally at line 869, right after the empty-jobs early return and before the per-page loop. On any real CLI run translator is None, so anthropic_translator() always executes. That function raises ConfigError (exit code 2) if the anthropic package is missing ('the anthropic package is not installed; run with uv run --frozen --group translate' — the group is non-default, so normal installs don't have it) or if no ANTHROPIC_API_KEY is configured.

Why zero-call runs are a real, intended path. translate_page has a deliberate shortcut at lines 839-840: if not job.open: return _assemble(english, ids, job, english), with the comment 'every English section still has a recorded translation: reassemble, no call'. This path is hit when classify finds every current English section hash already recorded — i.e. English sections were only reordered or removed — producing status 'outdated' with changed=[], which select_jobs turns into a Job with open=[]. The test suite proves the path makes no model call: tests/docs/test_translations.py has test_removed_english_section_is_reassembled_without_a_model_call, which passes a FakeTranslator([]) (a translator with zero prepared completions) and succeeds.

Concrete walkthrough. A docs contributor deletes one ## section from an English page and wants to refresh the twelve languages' generated pages. Every selected job is a reassembly job: (1) classify records changed=[] for the page in every language; (2) select_jobs builds jobs with open=[]; (3) translate_page would return via the line-840 reassembly path for each, making zero API requests. But step 0 never completes: without the translate group installed (it's non-default) or without ANTHROPIC_API_KEY, line 869 raises ConfigError and the command exits 2 before the loop starts. The contributor must install an SDK and configure credentials that this run would never use.

Why nothing catches it. Every test injects a FakeTranslator via the translator parameter, so anthropic_translator() is bypassed in the entire suite; the eager construction is only exercised by real CLI runs.

Why the current code is understandable but improvable. The eager construction is a deliberate fail-fast (the code comments that credentials should fail 'before any page work, rather than on the first request'), which is the right behavior for runs that DO call the model. The proposed guard preserves exactly that property while unblocking the offline path:

if any(job.open for job in jobs):
    translator = translator or anthropic_translator()

(with translate_page never touching translator when job.open is empty, this is safe — a None translator is only reachable on jobs that don't use it; alternatively pass the check inside the loop or defer to the first complete() call).

Impact. This is a quality/ergonomics finding in a maintainer-facing script, not a correctness bug — the tool fails clearly with exit 2 rather than misbehaving, and all three independent verifiers graded it non-blocking. It's worth fixing because it defeats an offline capability the tool explicitly implements and tests.

Comment thread i18n/README.md
uv run --frozen python scripts/docs/translations.py stage --lang CODE
```

`status` is offline: per language it lists missing, outdated (with the sections that changed), current and removable pages (translations whose English page is gone — `git rm` them). `translate` calls the Claude API (`ANTHROPIC_API_KEY` in the environment; the registry's model, or `DOCS_TRANSLATE_MODEL` to trial another) for the missing and outdated pages, retranslating only the English sections that changed and keeping the rest byte for byte; `--pages` instead re-translates exactly the named pages from scratch, which is also how a glossary or instructions change reaches existing pages (each generated page records the English section hashes it reflects, so editing those inputs invalidates nothing). `stage` assembles the tree a language site is built from; `scripts/docs/build.sh` runs it for every language. Commit the generated pages in an ordinary pull request.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 In scripts/docs/translations.py (anchored here because the tool's contract is documented on this line), code-fence placement is validated only page-wide: _restore_fences compares fence counts across the whole page and restores English fences positionally, while links get per-section checks. A model reply that keeps the fence count but moves a fence across a ## section boundary is therefore (a) published silently with code under the wrong section on a fresh translation, and (b) on a partial update only caught by _assemble()'s post-carry-forward reimpose, which raises PageError from inside the retry loop — aborting the page with zero of the MAX_REPAIRS repair turns the same mistake gets on a fully-open page. Fix both by comparing fence counts per section (via fence_ranges over sections()) and routing the post-assembly Mismatch findings into the same repair_request() feedback path instead of raising.

Extended reasoning...

The gap

reimpose() (translations.py:669) copies the English code fences over the model's reply so translated pages can never corrupt code samples. But its fence check, _restore_fences (line 685), is page-wide and positional: it only verifies len(want) != len(got) over the whole page, then splices English fence i into the reply's i-th fence position — wherever the reply put it. Contrast _check_targets (line 717), whose docstring is explicit about the design principle: link targets are compared "section by section, so any assembly of passing sections passes too". The per-section principle was applied to links, inline code spans, and block markers, but not to fences — the one element whose content is restored positionally.

None of the other validations can catch a fence relocated across a ## boundary with headings unchanged: _pin_headings is a positional count/level match, code_spans() masks fenced blocks, markers() masks fences, and link_targets uses masked text (fences rarely contain markdown links).

Failure mode 1 — fresh translation: silent wrong content

Step-by-step proof (reproduced by the finder executing the module's own functions with zensical stubbed):

  1. English page: ## Install contains one console fence, ## Test contains one pytest fence. Total: 2 fences.
  2. The model's reply keeps both fences but places both under Install — a classic LLM reflow slip, exactly the failure class this repair/validation subsystem exists for.
  3. _restore_fences: reply has 2 fences, English has 2 → count check passes; English fences are spliced positionally into the reply's (wrong) positions. _pin_headings passes (same headings, same levels). _check_targets passes (no links moved). reimpose returns a clean str, zero findings.
  4. For a fresh job job.previous is None, so carry_forward is a no-op and _assemble's second reimpose sees the identical page that already passed. _validate_open returns [].
  5. translate_page writes the page: the pytest fence sits under Install, and Test ends with its lead-in sentence and no code at all. If the reply additionally swaps the two fences' order, positional restoration puts each English fence under the wrong prose.
  6. stage()/serve() rerun the same page-wide check, so the misplacement survives into the built site with no diagnostic.

Failure mode 2 — partial update: abort instead of repair turns

On an update job with one section carried forward, the same reply passes the first reimpose, but carry_forward() splices the carried section back from the previous translation (which still has its fence), so the assembled page has 3 fences vs 2. _assemble() (line 815-820) then raises PageError('3 code fences vs 2 in the English: keep each code block, add none') — from the body = _assemble(...) call at line 854, which sits inside the retry loop with no try/except. The exception propagates straight out: the page fails on the model's first reply with zero of the 1 + MAX_REPAIRS (= 3) attempts used, even though the findings text is literally phrased as an instruction to the model, and the identical mistake on a fully-open page would get fed back via repair_request(). (The failure is safe — command_translate keeps the previous translation and exits 1 — but the update flow loses the repair machinery for exactly the mistake class it was built for.)

Fix

Compare fence counts per section (fence_ranges over sections(), mirroring _check_targets), which turns case 1 into a repairable first-reimpose finding and makes case 2 surface before assembly; and route the post-assembly Mismatch from _assemble into the same repair_request() feedback path (e.g. return the Mismatch and handle it like the isinstance(result, Mismatch) branch) instead of raising, so a structural mistake that only surfaces after carry-forward still gets its repair turns. One caveat: the pre-loop _assemble call (the if not job.open reassembly path, where no model is in play) should keep raising.

Why nit

This is docs-translation tooling only — never shipped in the mcp package — and both cases require a structurally-wrong model reply. Case 1 is the sharper consequence (silent wrong-section code on the published translated site, and defending against exactly such replies is this subsystem's purpose), but nothing in the SDK breaks, nothing corrupt is written on the update path, and re-running translate --pages is a workaround. All six verifier confirmations agreed on nit.

Comment thread i18n/README.md
Comment on lines +11 to +16

```text
uv run --frozen python scripts/docs/translations.py status [--lang CODE]
uv run --frozen --group translate python scripts/docs/translations.py translate --lang CODE [--pages PATH ...]
uv run --frozen python scripts/docs/translations.py stage --lang CODE
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 stage() deletes only the staged docs tree at the top and writes titles.json only at the very end, so a stage run that fails partway leaves the previous run's titles.json behind — and build_config.py --lang treats that file's presence as "the tree beside it is complete". A subsequent manual build_config then either dies with a confusing nav-validation error or silently builds a half-staged tree. One-line fix: unlink the titles file next to the rmtree at the top of stage() (e.g. staged_titles_file(...).unlink(missing_ok=True)).

Extended reasoning...

The bug. stage() in scripts/docs/translations.py starts with shutil.rmtree(staged_docs_dir(...), ignore_errors=True) but never removes the sibling .build/i18n/<code>/titles.json; that file is only (over)written as the very last statement of a successful run. Meanwhile build_config.py --lang explicitly relies on the file's presence as a completeness signal — its own comment reads "written last by stage, so its presence means the tree beside it is complete" — and only emits the actionable cannot read ... (run translations.py stage --lang X) message when titles.json is absent. That invariant only holds for a virgin .build directory: after any prior successful stage, an interrupted re-stage leaves a stale titles.json next to a deleted or partial docs tree.\n\nThe code path. The failure window is real and not exotic. Translatable pages are classified before the rmtree, but excluded pages are read mid-loop via read_english(repo.docs / page) (which raises ConfigError on OSError), and the per-page loop renders ~40 pages per language — so a Ctrl-C, a write error, or a ConfigError on an excluded page can interrupt the run after the rmtree but before titles.json is rewritten. Nothing prevents this because the only cleanup is the docs-tree rmtree; the titles file simply isn't in its scope.\n\nImpact. After such a failure, build_config.py --lang X reads the stale titles instead of failing with the intended "run translations.py stage" message. Depending on how far stage got: (a) rmtree ran but copytree didn't → a confusing SystemExit from nav validation ("nav references pages that don't exist under .build/i18n/X/docs"); or (b) copytree finished but the per-page loop didn't → every nav page exists as a raw English copy, so validation passes and zensical build silently produces a half-staged tree: pages missing the injected translation notice, dead ../api/ links (the api/ dir is excluded from the copy and rewritten links are only produced in the loop), and nav titles from the previous run. CI is shielded — build.sh does rm -rf .cache site .build/i18n up front under set -euo pipefail — so only the manual per-language flow documented in this README section is affected, and nothing broken can ship.\n\nStep-by-step proof. 1) Run translations.py stage --lang de successfully → .build/i18n/de/docs/ and .build/i18n/de/titles.json both exist. 2) Edit some English pages, run stage again and Ctrl-C while it is mid-way through the per-page loop → the rmtree has already deleted the old tree, copytree has recreated it, but only some pages carry the injected notice and API-link rewrite; titles.json from step 1 is untouched. 3) Run build_config.py --lang de → the open(titles_file) succeeds, so the guard that would have printed "run translations.py stage --lang de" never fires; nav validation passes because every page exists as a raw English copy; zensical build produces a de site where unprocessed pages have no translation notice, broken relative ../api/ links, and sidebar titles from the previous run.\n\nFix. One line at the top of stage(), next to the rmtree: staged_titles_file(language.code, repo.root).unlink(missing_ok=True). This restores the invariant the build_config comment documents — the titles file exists only when the tree beside it was fully staged.\n\nSeverity. This is docs-build tooling, not shipped SDK code; the trigger requires an interrupted manual re-stage followed by running build_config without re-staging, recovery is simply re-running stage, and CI is unaffected — hence a nit rather than a blocker.

Comment thread i18n/README.md
Comment on lines +13 to +15
uv run --frozen python scripts/docs/translations.py status [--lang CODE]
uv run --frozen --group translate python scripts/docs/translations.py translate --lang CODE [--pages PATH ...]
uv run --frozen python scripts/docs/translations.py stage --lang CODE

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 scripts/docs/translations.py re-declares two subtle regexes that already exist in llms_txt.py, from which it already imports: CODE_SPAN is byte-identical to llms_txt._CODE_SPAN, and FRONT_MATTER differs from llms_txt._FRONTMATTER only in the capture-group name. Consider exporting the two patterns from llms_txt (or a small shared module) and importing them, as page_url already is, so a future fix to either pattern cannot silently diverge between the translation tool and the llms.txt pipeline.

Extended reasoning...

What is duplicated. scripts/docs/translations.py (new in this PR) defines two module-level regexes that are private copies of patterns in the pre-existing sibling scripts/docs/llms_txt.py:

  • CODE_SPAN (translations.py:83) is byte-identical to llms_txt._CODE_SPAN (llms_txt.py:80):
    (?s)(?<!)(+)(?!)((?:(?!\n[ \t]*\n).)+?)(?<!)\1(?!)`
  • FRONT_MATTER (translations.py:69) is identical to llms_txt._FRONTMATTER (llms_txt.py:82) except for the capture-group name (body vs block), with the same re.MULTILINE | re.DOTALL flags.

Why it matters. These are not incidental patterns — both encode load-bearing parsing rules that mirror how the built site parses pages. The code-span regex implements CommonMark's backtick-run pairing ((?<!)(+)(?!)...\1) with a blank-line bound that stops a stray backtick from swallowing paragraphs; the front-matter regex deliberately mirrors mkdocs.utils.meta's block delimiting (---opener,---or...` closer), per llms_txt's own comment. Any future correction — e.g. a tightening of the kind the CodeQL alert on this very PR requests for a neighbouring regex in the same file, or an adjustment tracking a Zensical/MkDocs parsing change — will naturally land in one module and silently leave the other behind. At that point the translation tool and the llms.txt pipeline disagree about what counts as a code span or front matter, which is exactly the class of quiet drift that is hard to notice because both tools keep "working".

Why the reuse is cheap. translations.py already imports from the sibling module (from llms_txt import page_url, line 39), so sharing the two patterns extends an existing dependency rather than creating a new coupling. The fix is to promote the two patterns to importable names in llms_txt (or a tiny shared module under scripts/docs/) and import them; the capture-group name can be unified in the process since both call sites access the group.

Scope check. The duplication claim is deliberately narrow: the two modules' _FENCE regexes genuinely differ (translations.py captures the backtick run and info string and tracks closers; llms_txt only matches openers), so those are per-module by design and are not part of this finding. Only the two named patterns are semantically identical copies.

Concrete divergence walkthrough. Suppose a future PR fixes the front-matter pattern in llms_txt.py to also accept a UTF-8 BOM before the opening --- (a real MkDocs behaviour via mkdocs.utils.meta). llms_txt.py then extracts front matter from a BOM-prefixed page while translations.py's FRONT_MATTER.match at line 124 fails on the same file — so the translation tool treats the provenance block (translation.sections hashes) as body text, mis-hashing the page and re-translating sections that did not change. Nothing errors; the two pipelines just quietly compute different answers from the same file.

No runtime misbehaviour today — the copies are currently identical, so this is purely a maintenance/reuse cleanup and does not block merging.

Comment thread i18n/README.md
- `languages.yml` — the registry: one entry per translated site (served at `/<code>/`), the model id, and the nav pages that stay in English.
- `general-prompt.md` — translation rules shared by every language. `notices.md` — English source of the three notes staged onto the pages of a translated site.
- `<code>/instructions.md` (register, voice, typography, terminology) and `<code>/glossary.json` (`keep`: terms that stay in English; `terms`: required renderings, each with an optional `note` and banned `avoid` renderings, which are checked) — human-authored, sent with every request.
- `<code>/pages/**` and `<code>/notices.md` — **generated**, never edited by hand: a correction goes into that language's `instructions.md` or `glossary.json` (or the English page), and the affected pages are re-run.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The 200 generated pages under i18n/ja/, i18n/ko/, i18n/pt/ and i18n/zh/ (49 pages + notices.md per language) still carry a legacy inputs: <hash> front-matter key from intermediate commit c65a026, which the shipped tool's with_provenance() no longer writes — the other eight languages have no such key. Nothing breaks (read_provenance() tolerates it), but half the committed corpus isn't reproducible by the shipped tool, and the first translate run touching each of these pages silently drops the key, so a one-line English edit produces an extra unrelated front-matter diff. Mechanical fix: rewrite those files' front matter into the shipped format before merge (no API calls needed).

Extended reasoning...

What the inconsistency is. The generated-file contract this PR ships is defined by with_provenance() in scripts/docs/translations.py (~line 261): it writes exactly {translation: {sections: [...], tool: 1}} into each generated page's front matter. But the four languages published in intermediate commit c65a026 (Publish translated docs (zh-CN, ja, ko, pt-BR)...) were generated by an earlier version of the tool that also recorded an inputs: <hash> key. Commit 6107a94 then trimmed the tooling, dropping that key from what the tool writes — but the committed pages were not regenerated. Verified in the PR head: grep -rl ' inputs: ' i18n/<lang> finds exactly 50 files each under i18n/ja, i18n/ko, i18n/pt and i18n/zh (49 pages plus each language's notices.md; pages/translations.md in those languages is new and lacks it), and zero under de, es, fr, hi, ru, tr, uk and zh-hant.\n\nWhy nothing catches it. read_provenance() (line ~272) parses the front matter and reads only record['sections'] and record['tool']; the extra inputs key is silently ignored. Both old and new files carry tool: 1, so TOOL_VERSION — which exists precisely to mark generated-file-contract changes — treats the stale pages as current, and status never flags them. Nothing forces regeneration.\n\nThe concrete cost — a step-by-step example. Take i18n/ja/notices.md, whose front matter today reads:\n\nyaml\ntranslation:\n sections: [aff1b3e872b7876a, 4d80558ad052d586, 0bb81f1e62062d26, d5c35dcec50156bc]\n inputs: 8113dbdcdf8cc017\n tool: 1\n\n\n1. A maintainer makes a one-line edit to one section of the English i18n/notices.md.\n2. status reports the ja page outdated (one section hash changed); translate --lang ja re-translates only that section and carries the rest forward byte for byte — as designed.\n3. The tool then rewrites the whole file via with_provenance(), which emits only sections and tool. The inputs: 8113dbdcdf8cc017 line vanishes.\n4. The resulting diff is the intended one-line translated change plus an unrelated front-matter deletion — contradicting the PR description's stated property that "a one-line English edit is a one-line translated diff", for the first update of each of these 200 pages.\n\nWhy it matters beyond diff noise. Half of the committed generated corpus is not byte-reproducible by the tool this PR ships, even though the repo's own contract (i18n/README.md, and CONTRIBUTING.md's "never edit the generated pages" rule) presents these files as pure tool output. A reviewer diffing a future translation PR will see front-matter churn they can't attribute to the change under review.\n\nHow to fix. Purely mechanical, no API calls: strip the inputs: line from the 200 affected files (or equivalently re-serialize their front matter through with_provenance() with the existing section hashes). The translated bodies and section hashes are untouched, so status output is identical before and after.\n\nSeverity. This is a code-quality/consistency finding: nothing fails at build, stage or translate time, and readers of the published sites are unaffected. It should not block merging — hence nit. (The original finding said 204 files; the actual count is 200, which doesn't change the substance.)

Comment on lines +11 to +18
document$.subscribe(() => {
let page = location.pathname.slice(site.length);
if (page.startsWith("api/")) page = "";
for (const entry of document.querySelectorAll(".md-select__link[hreflang]")) {
entry.dataset.site ??= entry.getAttribute("href"); // the language root the theme rendered
entry.href = entry.dataset.site + page;
}
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The language switcher rewrites each entry using only location.pathname, so location.hash (and location.search) are dropped — a reader deep-linked to a section like /servers/tools/#structured-output who switches language lands at the top of the target page. Since the translation pipeline pins identical heading ids across all language sites, appending the hash (read at click time) would keep the reader on the same section.

Extended reasoning...

What the bug is. docs/js/language-switch.js computes the page path as location.pathname.slice(site.length) (line 12) and rewrites each language-switcher entry as entry.href = entry.dataset.site + page (line 16). location.hash and location.search are never appended, so the section anchor is lost whenever a reader switches language.

The code path. A reader lands on, say, /servers/tools/#structured-output (a deep link shared in an issue or chat). The document$.subscribe handler fires, rewrites every .md-select__link[hreflang] entry to <language-root>servers/tools/, and the reader who then picks another language arrives at the top of the target page instead of the #structured-output heading.

Why the fragment would actually work if preserved. This isn't a case where the anchor might not exist on the other side. scripts/docs/translations.py deliberately pins every translated heading with the English renderer's id — Repo.heading_ids (line 377) feeds _pin_headings (lines 703–713), which rewrites each heading as ## <translated text> {#<english-id>}. The committed generated pages visibly carry these attr blocks (e.g. # Übersetzungshinweise {#translation-notices}, ## Zusammenfassung {#recap} in the i18n/de pages). So the exact same fragment resolves to the same heading on every language site; the pipeline goes out of its way to make anchors portable, and the switcher then throws them away.

Why nothing else prevents it. The script's own header comment states its purpose is to keep the reader "on the current page" when changing language; the theme default (link to each language's home page) is what this script replaces, so there's no other layer that would restore the position. The api/ special case is unaffected since page is reset to the site root there anyway.

Step-by-step proof.

  1. Reader opens https://<site>/servers/tools/#structured-output — browser scrolls to the Structured output heading.
  2. document$ fires; page = "servers/tools/" (pathname only). Each switcher entry becomes e.g. https://<site>/de/servers/tools/.
  3. Reader clicks "de - Deutsch". Browser navigates to /de/servers/tools/ — no fragment.
  4. The German page does contain {#structured-output} on the corresponding heading (pinned by _pin_headings), but the browser was never told to scroll there. Reader lands at the top.

How to fix. Append the current hash (and search) to the rewritten href. One nuance the verifiers noted: the rewrite runs at document$ time, so naively doing entry.href = entry.dataset.site + page + location.hash captures only the load-time hash — a reader who clicks to a different section before switching would carry a stale fragment. The robust fix reads the hash at click time, e.g. a click handler on the entries that sets entry.href = entry.dataset.site + page + location.search + location.hash just before navigation (or recomputes in a click listener).

Severity. Nit: nothing breaks — the reader still reaches the correct page in the correct language (already better than the theme default), just scrolled to the top. All three verifiers independently confirmed the facts and agreed this is a real but non-blocking UX polish item.

Comment thread docs/translations.md
Comment on lines +13 to +16
If a translated page and its English original disagree, the English page is correct. Every page of a translated site opens with one of three notes saying where it stands:

- **Machine translation** — the page was translated automatically and links to its English original.
- **Translation behind the English page** — the English original changed after the page was translated, so parts of it may be out of date until the translation catches up.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 A pure reorder of ## sections on an English page makes classify() keep the stale translation (status='outdated', changed=[]), and serve() in scripts/docs/translations.py then reimposes today's English onto that stale body purely positionally — splicing each code fence under the wrong section's prose and cross-wiring the pinned heading anchors ({#install}/{#test} swapped) — instead of the documented 'served in English until refreshed' fallback. Fix: reorder the stored sections by their recorded provenance hashes before reimposing (the exact mapping carry_forward() already does in the translate path), or fall back to English when tuple(hashes) != recorded beyond the changed set.

Extended reasoning...

What happens

When an English maintainer cut-pastes a ## section up or down (a pure reorder — section bytes unchanged, only order changed), classify() (scripts/docs/translations.py:494-497) computes changed = [index for index, value in enumerate(hashes) if value not in set(recorded)]. Every current hash is still in set(recorded), so changed=[], and the page gets status='outdated' with note 'English sections removed or reordered' — and the stale translation is retained.

serve() (translations.py:889-900) then calls reimpose(state.english, repo.heading_ids(state.english), state.translation.body) directly on the stale body, which is still in the old section order. reimpose pairs everything positionally: _restore_fences splices today's English fence i into the translation's fence-i slot (counts match after a swap: each section still has exactly one fence), and _pin_headings pins the renderer's ids positionally (heading counts and levels match). _check_targets is the only per-section check, and it passes whenever the swapped sections carry no distinguishing links. No Mismatch is raised, so the English fallback never triggers.

Step-by-step proof (reproduced by multiple independent verifiers executing the module's own functions)

  1. English page: [intro, ## Install (uv add fence), ## Test (pytest fence)]; stored German translation recorded against those hashes.
  2. Swap the two sections in English (bytes per section unchanged — sections() assigns pre-heading blank lines to the following section, so section bytes are order-independent).
  3. classify()status='outdated', changed=[], translation kept.
  4. serve()reimpose returns a str (not Mismatch) containing:
    • ## Installieren {#test} with prose "Installiere es:" followed by the uv run pytest fence
    • ## Testen {#install} with prose "Fuehre die Tests aus:" followed by the uv add mcp fence

Every code block is published under the wrong prose, and every in-site link to #install/#test scrolls to the wrong section — staged silently with only the mild 'outdated' notice (stage prints nothing; language builds are non-strict).

Why this matters

This breaks the PR's own degradation contract — docs/translations.md and the PR description promise that a translation that no longer fits its English page is "served in English until refreshed." In exactly the state the system labels 'outdated', it serves cross-wired content instead. Exposure is real: scripts/docs/build.sh runs stage for every language on every docs deploy, and the design deliberately deploys English-only PRs against stale translations, so every deploy between the reorder merge and the next translate run ships the corrupted page in all 12 languages. A reorder combined with an edit ([i,A,B] → [i,B,C]) triggers the same cross-wiring while status only reports the edited section.

Why nothing existing prevents it

The correct algorithm already exists three functions away: on the identical state, the translate path short-circuits (job.open == []) into carry_forward(), which maps sections by recorded hash (prior = dict(zip(previous.sections, sections(previous.body)))) and emits them in the new English order — verified to produce a fully coherent page with zero model calls. serve() simply ignores the provenance order it has in hand. (This is distinct from the per-section fence-count concern raised elsewhere: after a pure swap each positional pair still has exactly one fence, so counts match while contents are wrong, and no model reply is involved at all.)

Fix

In serve(), before reimposing, reorder the stored translated sections by their recorded hashes against today's section_hashes(english) (reusing carry_forward's mapping), or refuse and fall back to English when tuple(hashes) != recorded beyond the changed set — which is the documented behavior. Small, localized change; the hash-keyed mapping is already proven in the same file.

Comment thread i18n/README.md
Comment on lines +11 to +15

```text
uv run --frozen python scripts/docs/translations.py status [--lang CODE]
uv run --frozen --group translate python scripts/docs/translations.py translate --lang CODE [--pages PATH ...]
uv run --frozen python scripts/docs/translations.py stage --lang CODE

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Commit d37ff04's rewrite of HEADING_ATTRS (scripts/docs/translations.py:76) does not fix the CodeQL-flagged exponential backtracking: inside each repeated block, :? and the body class [^}\n]* can both consume a :, so a heading with colon-bearing blocks (the canonical {: #id} attr_list spelling) followed by trailing text takes 2^n time — ~13s at 24 blocks, effectively forever at ~40 — and the regression test added in the same commit only covers the colon-free shape that was never ambiguous. Fix by making each block single-parse (drop :? from the pattern and lstrip one leading : from the captured body in parse_headings), and extend the test to a {: #a}-shaped run with trailing text.

Extended reasoning...

What the bug is. HEADING_ATTRS = re.compile(r"(?:[ \t]*\{:?(?P<body>[^}\n]*)\})+[ \t]*$") at scripts/docs/translations.py:76 is still exponentially ambiguous after commit d37ff04 ("Simplify the heading attribute pattern", the head of this PR), which was written specifically to resolve the CodeQL inefficient-regex alert on this line. Inside each repeated unit, the optional :? and the body class [^}\n]* (which excludes only } and newline) can both consume a :. Every colon-bearing block therefore has two parses that converge on the same end position, and when the overall $-anchored match fails — a run of blocks followed by trailing text — the engine explores 2^n paths. Note that {: #id} with the colon is the canonical attr_list spelling; the :? exists precisely to accept it, so the pathological shape is the expected input, not a corner case.\n\nMeasured on this PR head with the actual compiled pattern: 'x ' + '{: #a}'*n + ' tail' takes 0.08s / 0.30s / 1.14s at n = 16 / 18 / 20 — a clean 4× per two extra blocks, i.e. 2^n — reaching ~13–18s at 24 blocks and effectively hanging forever near 40. Colon-free shapes ('{}'*24, '{ a } '*20) are instant (~20µs) because they are unambiguous.\n\nThe guarding test certifies the wrong shape. The regression test added in the same commit, test_many_attribute_blocks_pin_the_last_id_only_when_they_end_the_heading (tests/docs/test_translations.py:295), uses blocks = "{ a } " * 20 — exactly the colon-free shape that was never ambiguous — while its docstring asserts "the scan still returns promptly however many blocks it has." The test passes, and the commit message's claims ("could take a very long time … Parsed results are unchanged", "covered by a new unit test") read as if the property holds, while the pathological class survives. The CodeQL alert posted at 14:14 — after the head commit at 14:11 — flags the shipped pattern, so the alert remains live against this PR.\n\nConcrete trigger path. parse_headings() (translations.py:174) runs HEADING_ATTRS.search on every heading line of every model reply via translate_page → reimpose → _pin_headings, and the pattern's own comment (lines 73–75) says it is deliberately loose so blocks "the model glued to CJK text or doubled are still seen" — degenerate replies are anticipated input. Repetition loops are a known LLM failure mode: a glitched reply containing ## Titel {: #a}{: #a}…{: #a} weiterer Text with ~30–40 repeated blocks and trailing text hangs the translate run inside the regex engine with no timeout. Nothing upstream prevents this: the hang happens during the very first parse of the reply, before MAX_REPAIRS, the Mismatch/repair turns, or any fence check can run. stage/status parse committed tool-written pages with the same pattern, so the live exposure is the model-reply path.\n\nStep-by-step proof. (1) A model reply for a page contains a heading line x {: #a}{: #a}…{: #a} tail with 24 colon-bearing blocks and trailing text. (2) parse_headings calls HEADING_ATTRS.search on the heading text. (3) The trailing tail defeats the $ anchor at every start position, so the engine backtracks; for each of the 24 blocks it tries both the :?-consumes-colon and body-consumes-colon parses — 2^24 ≈ 16.7M paths, ~13s measured. (4) At ~40 blocks (2^40 paths) the run never returns; the maintainer must kill the process and loses the run.\n\nHow to fix. Make each block single-parse: drop :? from the pattern — (?:[ \t]*\{(?P<body>[^}\n]*)\})+[ \t]*$ — and lstrip one leading : from the captured body in parse_headings before splitting. Parsed results are unchanged: the id extraction (line 175) whitespace-splits and keeps only #-prefixed tokens, so a bare : token is already ignored, and lstripping keeps glued {:#id} working. Then extend the regression test to a '{: #a}'-shaped run followed by trailing text, which currently would hang and after the fix returns instantly.

Comment thread CONTRIBUTING.md
Comment on lines +129 to +131
## Documentation and Translations

Documentation contributions are English only: the pages under `docs/` are the source of truth, and the translated documentation sites are generated from them, guided by the per-language style guides and glossaries under `i18n/<lang>/`. Never edit the generated pages under `i18n/<lang>/pages/`—the next translation run overwrites them. To fix a translation, change that language's `instructions.md` or `glossary.json` (or the English page, if that's where the problem is), and the fix carries into every future run. See [`i18n/README.md`](i18n/README.md) for the details.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The rationale here is inverted: the tool never overwrites a hand-edited generated page — classify() compares only the recorded English section hashes (never the translated body), so an edited page whose English is unchanged stays 'current' and is never re-run, and when the English later changes elsewhere, carry_forward() reproduces the hand-edited text byte-for-byte as fresh tool output, with no provenance field that could ever detect it. Reword this sentence to state the real consequence (a hand edit silently persists, is carried forward, and escapes the pipeline's validators); optionally also record a short hash of the translated body in the provenance block so status can flag diverged pages.

Extended reasoning...

What the line claims vs. what the tool does. CONTRIBUTING.md line 131 (added by this PR) justifies the never-edit rule with: "Never edit the generated pages under i18n/<lang>/pages/—the next translation run overwrites them." The actual behavior in scripts/docs/translations.py is the opposite: hand edits are sticky, not transient, and nothing in the pipeline can detect them.

The code path. select_jobs() (translations.py:772-786), absent --pages, selects only pages whose status != 'current'. classify() (477-497) computes status purely by comparing the recorded English section hashes in the page's front matter against the current English page — the translated body's content is never hashed or inspected (only its section count is checked, at line 489, which a typical in-section wording edit does not trip). A hand-edited page whose English hasn't changed therefore remains current indefinitely, and no translation run ever touches it. The "next translation run" overwrites nothing.

Why it gets worse, not better. When the English does later change in some other section, carry_forward() (802-812) maps each unchanged section through prior = dict(zip(job.previous.sections, sections(job.previous.body))) — i.e. it reproduces the previous on-disk translation, hand edit included, byte-for-byte. The edit is thereby laundered into fresh tool output. And _validate_open() only validates the sections rewritten in that run, so the carried hand-edited text also permanently escapes the pipeline's own checks (glossary avoid-list, code-span/marker parity). The only events that remove the edit are a deliberate translate --pages <that page> or an English change to the very section that was edited.

Why nothing detects it. with_provenance()/read_provenance() (260-274) record only the English section hashes and TOOL_VERSION — no digest of the translated body — so status reports a hand-edited page as current, and no command can later tell which of the ~600 committed pages are still genuine tool output. That silently breaks the property the PR's own contract depends on (i18n/README.md: "generated, never edited by hand"; docs/translations.md: "The generated text is never edited by hand"). Notably, i18n/README.md states the policy without the false mechanism claim, so the inaccuracy is specific to this CONTRIBUTING paragraph.

Concrete walkthrough. (1) A contributor spots a wording problem in i18n/de/pages/servers/tools.md and, trusting CONTRIBUTING's "it gets overwritten anyway," edits the prose directly (or resolves a merge conflict by hand). (2) classify() re-hashes the English page, finds tuple(hashes) == recorded, returns current — the page is never selected. (3) Months later a different ## section of the English page changes; the page becomes outdated, translate runs, and carry_forward() copies the hand-edited section verbatim into the regenerated file. (4) The edit is now indistinguishable from tool output, was never checked against the glossary or parity validators, and status has no signal that could ever surface it.

Why this matters. The rule itself ("never edit") is correct — only its stated enforcement is inverted. A reviewer who believes hand edits are futile will accept them casually ("it'll be regenerated anyway"), which is exactly how the corpus forks from tool output permanently and undetectably.

How to fix. Cheapest: reword the sentence to state the true consequence, e.g. "—a hand edit silently persists, is carried forward byte-for-byte into future regenerations, and escapes the pipeline's validators." Mechanically better: also record a short hash of the translated body in the provenance block (the front-matter format is tool-owned and TOOL_VERSION exists precisely to version such contract changes) so classify()/status can flag pages whose body no longer matches what the tool wrote.

Severity. All four verifiers confirmed every mechanism claim and agreed this is a documentation-accuracy issue plus a missing guard: nothing malfunctions at runtime today, so it should not block merge — nit.

Comment thread i18n/README.md
Comment on lines +16 to +18
```

`status` is offline: per language it lists missing, outdated (with the sections that changed), current and removable pages (translations whose English page is gone — `git rm` them). `translate` calls the Claude API (`ANTHROPIC_API_KEY` in the environment; the registry's model, or `DOCS_TRANSLATE_MODEL` to trial another) for the missing and outdated pages, retranslating only the English sections that changed and keeping the rest byte for byte; `--pages` instead re-translates exactly the named pages from scratch, which is also how a glossary or instructions change reaches existing pages (each generated page records the English section hashes it reflects, so editing those inputs invalidates nothing). `stage` assembles the tree a language site is built from; `scripts/docs/build.sh` runs it for every language. Commit the generated pages in an ordinary pull request.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Every docs build re-renders the identical English corpus 12 times: scripts/docs/build.sh runs one translations.py stage --lang X Python process per language, and each run recomputes classify() (re-reading + sha256-hashing all ~51 English pages) and Repo.heading_ids() (the full python-markdown render over every English body) — ~11s of byte-identical recomputation plus 11 redundant process startups per deploy/CI/preview build. Letting one invocation stage all languages (the way status already defaults to the whole registry when --lang is omitted) plus a body-keyed memo on heading_ids would compute the English hashes and heading ids once per build.

Extended reasoning...

What is duplicated. scripts/docs/build.sh (lines 65–74) builds the language sites with a shell loop that invokes uv run … translations.py stage --lang $lang once per language — 12 separate Python processes, each independently paying interpreter startup, zensical/markdown imports, mkdocs.yml parsing, and renderer construction. Inside each process, stage() (translations.py:928) calls classify() (line 477) for every page — which read_english()s and sha256-hashes the English source — and serve() (line 889) calls repo.heading_ids(state.english), which runs the full python-markdown pipeline (renderer.reset(); renderer.convert(body), lines 383–384) over every English page body.

Why 11 of the 12 renders are pure recomputation. heading_ids(body) is a pure function of the English body given the shared renderer config, and the English bodies are byte-identical across all 12 languages — the per-language inputs (instructions.md, glossary.json, the generated translations) never enter this computation. So every build computes exactly the same set of heading ids 12 times, and hashes exactly the same English files 12 times. Measured on this checkout with the repo's own code, the heading_ids pass alone is ~1.0s per language for the 51 pages — roughly 11s of duplicate compute per build, on top of the 11 redundant process startups, and the cost recurs on every deploy (deploy-docs.yml), every docs CI run (shared.yml runs build.sh), and every preview build (docs-preview.yml).

Step-by-step walkthrough. (1) build.sh computes the registry's 12 language codes and enters its loop. (2) Iteration de: a fresh Python process imports the translation tooling, parses mkdocs.yml, constructs the renderer, then stage() classifies all ~51 pages (read + hash each English source) and serve() renders each English body through the markdown pipeline to pin heading ids. (3) Iteration es: a second fresh process repeats every one of those steps against the same English bytes, producing the same hashes and the same ids. (4) Steps repeat for the remaining 10 languages. Only the per-language overlay work (which translation to serve, the injected notice, the staged tree) actually differs between iterations.

Why the fix is small and behavior-neutral. The proposed shape already exists in the file: command_status defaults to repo.registry.languages when --lang is omitted, while the stage subparser marks --lang as required=True (translations.py:1001). Making stage without --lang iterate the registry the same way, and memoizing Repo.heading_ids by body (a dict or functools cache on the Repo instance — the renderer is already shared and reset per call), means each English page is read, hashed and rendered once per build. stage() is per-language independent (it reads only English plus that language's generated pages), and build.sh already serializes the stagings under set -euo pipefail (any failure aborts the whole build either way), so a single-process loop changes no observable behavior; build.sh's loop shrinks to build_config + zensical per language.

Why this is a nit. Nothing is incorrect today — the output is right, and the PR description explicitly budgets "about a minute" of extra build time for the language sites. This finding just names the fraction of that minute that is byte-identical recomputation and the one-process shape that removes it. Worth doing since the cost is paid on every docs deploy, CI run, and preview, but it should not block merging.

Comment thread docs/translations.md
Comment on lines +19 to +21
## How the translations are made

Translated pages are machine-generated by a tool in this repository from the English pages under `docs/`, guided by two human-written inputs per language: a style guide (register, tone, typography, how to handle jokes and idioms) and a glossary (which terms stay in English, and the required and forbidden renderings for the rest). The generated text is never edited by hand. Every improvement goes into those inputs instead, so it survives the next time the pages are regenerated.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The abridgement check in scripts/docs/translations.py — the only validator guarding against a model reply that silently omits prose — matches hard-coded English placeholder phrases (_PLACEHOLDERS/_ABRIDGED, lines ~92–98), yet every reply the tool validates is in one of the twelve non-English target languages, so a shortened list followed by e.g. (Die übrigen Punkte sind sinngemäß identisch.) or (以下省略) passes reimpose() and validate() cleanly and is published; the page then hashes as current and carry_forward() freezes the abridged section into all future output, exempt from re-validation. Consider adding structural parity checks (per-section list-item/table-row/block counts vs the English, routed into the existing repair_request() loop) and/or per-language placeholder phrases via the glossary files.

Extended reasoning...

The gap. validate() (scripts/docs/translations.py:734) is the only defence against a model reply that omits prose — everything structural (fences, headings, code spans, links, !!!/=== markers) is separately re-imposed or counted by reimpose(). Its abridgement check, abridgements() (line 217), scans for three _PLACEHOLDERS regexes and an _ABRIDGED comment-word list (lines 92–98) that are hard-coded English phrases: [translation|rest of|remaining ...], (omitted...), [...], and comment words omitted|continues|truncated|abridged|remaining|rest of. But by construction every reply this tool validates is in one of the twelve non-English target languages — a model that cuts a page short writes its placeholder in the language it is writing, and none of the English patterns fire. The guard is calibrated for exactly the failure it will never see.\n\nStep-by-step proof (reproduced end-to-end by executing the module's own functions). Take an English page whose ## Common errors section contains a 7-item prose bullet list (no fences, links, or code spans in the items). The model's reply is a faithful German translation of everything except that list, which it cuts to 2 items followed by (Die übrigen Punkte sind sinngemäß identisch.) ("the remaining points are essentially identical"). Then: (1) reimpose() returns a clean str — heading counts/levels match, fence counts match, per-section link targets match, because the dropped items carried none of those. (2) validate() returns [] — code spans, block markers, and banned glossary renderings all pass, and abridgements() finds nothing because the placeholder is German. (3) translate_page accepts the body, no repair turn is spent, and command_translate writes the page with no diagnostic anywhere. Direct probes confirm the asymmetry: abridgements() catches '[rest of the page unchanged]', '(omitted for brevity)', '[...]', and '<!-- remaining sections omitted -->', but returns empty for '[Rest der Seite unverändert]', '(以下省略)', '<!-- 以下は変更なし -->', and '[el resto de la página sin cambios]'.\n\nNo other validator covers prose omission. i18n/general-prompt.md promises "the same lists with the same nesting and number of items, the same tables with the same rows and columns ... the same blank lines between blocks" — but list-item counts, table rows, and paragraph/blank-line structure are checked nowhere. This also means a shortened list with no placeholder at all escapes detection in any language, including English; the placeholder patterns are the tool's only line, and they only speak English.\n\nWhy it compounds. Once accepted, the page records current section hashes, so status reports it current forever — nothing ever flags it for re-translation. On every future update of other sections, carry_forward() reproduces the abridged section byte-for-byte, and _validate_open (line ~823) deliberately exempts carried sections from validation ("a carried section is published text, not this run's to fix"). The omission is frozen into all future tool output, invisible to status, stage, and reviewers of later translation diffs, who see only the changed sections. The published page shows the abridged section under the reassuring "Machine translation" notice rather than the "outdated" one.\n\nThis is not an intentional non-goal. The tool dedicates three regexes, a comment scan, and a MAX_REPAIRS repair loop to exactly this failure class — models demonstrably abridge, which is why general-prompt.md line 10 says "Never summarise, abridge, or leave a placeholder". The defence exists; it just cannot fire on the languages the replies are written in. (One mild mitigation: [...] is language-neutral, and models sometimes leave English meta-comments even when writing other languages — so the patterns are not useless, but the gap is real.) This is distinct from the previously reported fence-placement and section-reorder findings, which concern positional restoration of structural elements; this concerns prose-level omissions no structural check can see.\n\nFix directions (any one suffices): (a) validate the structure the prompt already promises — compare per-section list-item counts, table-row counts, and block counts between the English and the reply (both sides are already parsed line-wise), routing findings into the existing repair_request() loop exactly as spans/markers/links do; (b) make the placeholder check language-aware, e.g. flag a short bracketed/parenthesised line in a reply section whose English counterpart has list items the reply lacks; (c) supply per-language placeholder phrases via each language's instructions.md/glossary.json. Option (a) is the robust one, since the placeholder is optional but the dropped items are not. Severity-wise this is non-blocking: it is a maintainer-facing docs tool, the failure requires model misbehaviour, nothing in the SDK or the docs build breaks, and the published pages carry a machine-translation caveat with a reporting path — but the fix is cheap and enforces a contract the prompt already states.

Comment thread docs/translations.md
Comment on lines +9 to +17
The API reference is not translated: the translated site links to the single English one.

## English is the source of truth

If a translated page and its English original disagree, the English page is correct. Every page of a translated site opens with one of three notes saying where it stands:

- **Machine translation** — the page was translated automatically and links to its English original.
- **Translation behind the English page** — the English original changed after the page was translated, so parts of it may be out of date until the translation catches up.
- **Shown in English** — there is no current translation of the page, so you are reading the English text.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The staged language pages embed root-absolute URLs — render_notice() fills the notice's ENGLISH_PAGE/TRANSLATIONS_PAGE placeholders with /{page}/ and /{code}/translations/, stage() rewrites api/*.md links to /api/mcp/..., and build_config.py's nav API entry and language-switcher hrefs are likewise root-anchored — but scripts/build-docs.sh copies the whole site byte-identically to /v2/ with a documented contract that /v2/... links keep resolving after a future major takes the root. These are the first content links that escape the version prefix: today a reader on /v2/de/ clicking any of them silently leaves the pinned edition, and once a future major takes the root they point at the wrong major's pages (or 404). Fix by emitting prefix-relative links — stage() knows each page's directory depth, so ENGLISH_PAGE can be e.g. ../../../get-started/installation/, and the API rewrite, TRANSLATIONS_PAGE, nav entry and switcher can be computed the same way.

Extended reasoning...

What the bug is. Three places in the new translation tooling emit root-anchored absolute URLs into the staged/rendered language sites:

  1. render_notice() in scripts/docs/translations.py (~lines 905-906) fills the per-page notice placeholders with (ENGLISH_PAGE)(/{page_url(page)}) and (TRANSLATIONS_PAGE)(/{code}/translations/).
  2. stage() rewrites every api/*.md link in every staged page via the _API_LINK pattern (line 100, applied at line 948) into a root-absolute ](/api/mcp/... form.
  3. scripts/docs/build_config.py sets the language nav's API entry to API_REFERENCE_URL = "/api/mcp/" (line 42, used at line 208), and alternate() (lines 126-136) emits path-only switcher links "/", "/de/", … into every config, English included. Its docstring notes the links are deliberately path-only "so they follow whatever host serves the build" — host portability was considered, but path-prefix portability was not.

Leading-slash markdown link targets pass through the renderer verbatim (build_config's own _LINK classification treats them as links, not docs pages), so these paths land literally in the built HTML. Verified in the committed staged tree: .build/i18n/de/docs/get-started/installation.md carries [englische Seite](/get-started/installation/) and [Übersetzungen](/de/translations/), and index.md carries [API-Referenz](/api/mcp/).

Why it collides with the mirror contract. scripts/build-docs.sh — the unchanged deploy entry point run by deploy-docs.yml — copies the entire site/ tree, language subsites included, byte-identically to /v2/ (cp -a \$V2_WORKTREE/site/. \$OUTPUT_DIR/v2/), with an explicit documented promise (lines 5-9): per-major paths are permanent, and /v2/... links keep resolving after a future major takes the root, the way /v1/... does today. There is no rewrite pass. A grep of docs/**/*.md finds zero pre-existing root-absolute markdown links in English prose — the pre-PR mirror was fully self-contained (MkDocs/Zensical emit relative hrefs for .md targets), so this PR introduces the first content links that escape the version prefix.

Step-by-step example. (1) This PR merges; the deploy produces the root site and its byte-identical /v2/ copy, so /v2/de/get-started/installation/ exists and its notice link points at /get-started/installation/. (2) Today, root == v2, so clicking it merely (and silently) jumps the reader from the pinned /v2/ edition to the root edition — mildly wrong, nothing 404s. (3) A future major (v3) takes the root, exactly the event /v2/ exists for. Now /v2/de/get-started/installation/'s "English page" link points at v3's /get-started/installation/ (wrong content, or a 404 if the page moved), its "API Reference" nav entry at v3's /api/mcp/, and its language switcher at v3's /de/ — precisely the wrong-major breakage the mirror contract promises against, baked into content that is by design never regenerated.

Why nothing catches it. The tool's link-target validation compares translated targets against the English per section, and these URLs are injected after validation, at stage time; leading-slash targets are classified as external links and never existence-checked; the language builds are deliberately non-strict; and no CI step builds or crawls the /v2/ mirror.

How to fix. Emit prefix-relative links. stage() knows each page's directory depth, so ENGLISH_PAGE can be a relative path up to the sibling English page (e.g. ../../../get-started/installation/ from de/get-started/installation/), and the API rewrite and TRANSLATIONS_PAGE can be computed the same way; the nav API entry and the switcher hrefs need the same treatment (or, alternatively, the /v2/ mirror step in build-docs.sh could rewrite leading-slash hrefs to /v2/-prefixed ones).

Scope note. This is distinct from the hreflang alternate HEAD-metadata concern (root-vs-per-page alternate targets, an SEO-validity issue): per-page alternate targets would still be root-absolute and still break under the mirror, and relative content links do nothing for alternate self-reference — the fixes don't overlap. Severity: nothing misbehaves on the live site today, and the breakage only surfaces at the next major, so this shouldn't block merging — but it's worth fixing now, before these links ship into permanent, pinned /v2/ content.

Comment on lines +11 to +18
document$.subscribe(() => {
let page = location.pathname.slice(site.length);
if (page.startsWith("api/")) page = "";
for (const entry of document.querySelectorAll(".md-select__link[hreflang]")) {
entry.dataset.site ??= entry.getAttribute("href"); // the language root the theme rendered
entry.href = entry.dataset.site + page;
}
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The <link rel="alternate" hreflang> tags in every page's <head> come from build_config.py's alternate(), which emits root-only URLs (/, /de/, /ja/, ...), so every non-home page on all 13 sites declares its language alternates to be the site homepages and never references itself — an invalid hreflang cluster per Google's localized-versions rules, leaving the SEO metadata inert or misleading on every deep page. This script already solves the equivalent problem for the visible switcher (every prose page exists at the same path on all sites), but its selector never touches the head tags; either extend the rewrite to document.querySelectorAll('link[rel=alternate][hreflang]') or, better, emit per-page alternates statically via a small template override.

Extended reasoning...

What the bug is. The per-language registry gives each language a dedicated hreflang field (commented "announced in <link rel=\"alternate\">"), and alternate() in scripts/docs/build_config.py (lines 126-136) builds extra.alternate from it. But every entry it builds is a root-only link: / for English and /<code>/ for each language. The installed Zensical theme (0.0.53, templates/base.html lines 30-33) renders config.extra.alternate verbatim into every page's <head> as <link rel="alternate" href="{{ alt.link | url }}" hreflang="{{ alt.lang }}"> — identically on every page of the site.\n\nThe code path. Building the de site from this PR head, site/de/servers/tools/index.html carries <link rel="alternate" href="/" hreflang="en"> and <link rel="alternate" href="/de/" hreflang="de"> right next to <link rel="canonical" href="https://py.sdk.modelcontextprotocol.io/de/servers/tools/">. Per the hreflang contract (Google's localized-versions rules), each alternate must reference the equivalent page in the other language, and the cluster must include a self-referencing entry with reciprocal return tags. Here every non-home page across all 13 sites says "my English/German/... alternate is the site homepage" and never references itself — so only the homepage cluster is valid, and the annotations are broken or ignored on exactly the pages hreflang exists to disambiguate.\n\nWhy existing code doesn't prevent it. The PR already solved the equivalent problem for humans: language-switch.js rewrites the visible switcher entries to the current page, because every prose page exists at the same path on all sites. But its selector (line 15, .md-select__link[hreflang]) only matches the visible switcher anchors from partials/alternate.html — the <link rel="alternate"> tags in <head> are never rewritten. And even if they were, non-Google crawlers and any crawl phase that skips JS read only the static head, so a JS-only fix is partial by nature.\n\nStep-by-step proof.\n1. alternate() builds extra.alternate = [{name: 'en - English', link: '/', lang: 'en'}, {name: 'de - Deutsch', link: '/de/', lang: 'de'}, ...] — one entry per site, all root URLs.\n2. Zensical's base.html stamps those entries into the head of every rendered page, so /de/servers/tools/ gets hreflang="en" href="/" instead of href="/servers/tools/".\n3. A crawler evaluating the cluster for /de/servers/tools/ follows the en alternate to / (the English homepage), finds no return tag pointing back at /de/servers/tools/, and finds no self-referencing entry on the German page — the cluster fails validation and the annotation is discarded.\n4. language-switch.js runs at document$ time and rewrites only .md-select__link[hreflang] anchors; document.querySelectorAll('link[rel=alternate][hreflang]') would match the head tags, but the script never queries them.\n\nImpact. Nothing breaks for readers — the visible switcher works, the pages render, the build passes. The cost is that the hreflang machinery this PR ships is inert or misleading for exactly the audience (search engines) it exists for: the metadata that decides which language edition a search engine serves is only valid on the 13 homepages.\n\nHow to fix. Either (a) extend language-switch.js to also rewrite document.querySelectorAll('link[rel=alternate][hreflang]') with the same dataset.site + page logic (helps rendered crawls only), or (b) — better — emit per-page alternates statically via a small template override so the head is correct without JS, using the same invariant the switcher relies on: every prose page exists at the same path on every language site.\n\nNot a duplicate. The cubic comment on i18n/languages.yml line 32 concerns the value of the pt entry's hreflang code (pt vs pt-BR); this finding concerns the target of every alternate link on every page — region-qualifying pt fixes nothing here. The existing comment on language-switch.js line 18 concerns location.hash being dropped in the visible switcher (runtime UX for humans); this is the static SEO metadata crawlers read.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants