Share doctest namespaces without collapsing per-block node ids

[tony]

Summary

Four open issues — #83, #84, #85, #86 — all change how doctest_docutils turns a document into doctest.DocTest objects. Two of them, #83 (blocks do not share a namespace) and #85 (node ids embed an absolute path), pull toward the same implementation shortcut: emit one DocTest per document instead of one per block. That shortcut delivers both, and silently spends something neither issue mentions — per-block addressability.

Upstream Sphinx keeps per-block DocTest objects while sharing a namespace, which is an existence proof that the mechanism is sanctioned — though Sphinx is a builder with no node ids, no -k and no workers, so it demonstrates the mechanism, not that per-block addressability transports with it. This issue asks that the namespace work land the way Sphinx does it, so sharing state and keeping one item per block stay independent choices.

Motivation

A page collected today yields one item per block:

$ pytest docs/topics/automation_patterns.md --collect-only -q

16 tests collected

Those ids are what make a single block selectable. Losing them costs:

• pytest page.md::page.md[6] to re-run one block while iterating on it
• --lf re-running only the block that failed, rather than the page
• -k and --deselect reaching a block
• a JUnit report naming the block that failed

None of that is speed. Measured on a 16-block page, one item per block is in fact slower than one per document, because xdist worker startup outweighs the gain at that size:

Layout	Wall clock
16 items, -n auto	6.73 s, 6.38 s
serial (what 1 item costs)	4.65 s, 5.10 s

So the argument for keeping per-block items is debuggability and tooling, not throughput. Worth stating plainly, because a reader could otherwise assume the reverse.

Background: why one DocTest per document is the shorter path

DocTest.__init__ copies the namespace it is handed:

• cpython/Lib/doctest.py#L562 — self.globs = globs.copy()

and DocTestRunner.run empties it afterwards by default:

• cpython/Lib/doctest.py#L1504 — run(self, test, compileflags=None, out=None, clear_globs=True)
• cpython/Lib/doctest.py#L1577-L1578 — if clear_globs: test.globs.clear()

Between the copy and the clear, N DocTest objects cannot share state by default. Collapsing them into one is the shortest way around both.

gp-libs currently emits one per node:

• gp-libs/src/doctest_docutils.py#L385-L402 — the per-node loop
• gp-libs/src/doctest_docutils.py#L416 — get_doctest(string, globs, name, filename, lineno)

Proposal

Adopt Sphinx's mechanism: build a DocTest per block as today, then assign the shared mapping and disable the clear. Sphinx annotates both obstacles in-line:

• sphinx/ext/doctest.py#L525-L526 — one namespace per group, ns: dict[str, Any] = {}
• sphinx/ext/doctest.py#L600-L613 — # DocTest.__init__ copies the globs namespace, which we don't want → test.globs = ns, then # also don't clear the globs namespace after running the doctest → run(test, out=..., clear_globs=False)
• sphinx/ext/doctest.py#L538-L549 — the same pattern for testsetup / testcleanup

The stdlib sanctions it: DebugRunner passes clear_globs=False itself, at cpython/Lib/doctest.py#L1960-L1961.

This keeps one pytest item per block and gives blocks a shared namespace, which are the two things #83 and #85 want separately.

Compatibility

Additive on four axes, and constrained on a fifth:

• stdlib doctest — nothing subclassed or monkeypatched. clear_globs is a documented parameter of DocTestRunner.run and test.globs a documented attribute; the clear itself is Lib/doctest.py#L1577-L1578. The stdlib sanctions the override: DebugRunner.run passes clear_globs=False itself. One caveat worth recording: compileflags is derived from the namespace at run time via _extract_future_flags(test.globs), so a shared mapping also shares __future__ flags between blocks.

• _pytest.doctest — untouched. Its own text collector stays one-DocTest-per-file at _pytest/doctest.py#L423-L444; DoctestItem keeps taking a dtest per item at #L251.

• the doctest_namespace fixture — pytest merges it with self.dtest.globs.update(globs) at #L288-L293. Seeding the shared mapping before the first block preserves that.

• existing suites — default stays one namespace per block, so no suite changes behaviour until it opts in.

• pytest-xdist — this is the axis the issue originally omitted, and it governs. Each worker is a separate process that performs its own collection (docs/how-it-works.rst), so a shared mapping does not cross workers. Measured against a shared-mapping build on the two-block page from Doctest blocks in one document do not share a namespace, unlike pytest text files #83: serial 2 passed; -n 2 (default scheduler) 1 failed, 1 passed; -n 2 --dist loadfile 2 passed; -n 2 --dist loadgroup without an xdist_group marker 1 failed, 1 passed.

  --dist defaults to no, and -n without --dist promotes it to load at src/xdist/plugin.py#L304-L334. loadfile keeps a file's items on one worker by splitting the scope on the nodeid's path (scheduler/loadfile.py#L35-L60), which is the documented guarantee at docs/distribution.rst. loadgroup reads the xdist_group marker on the worker and suffixes the nodeid with @group (src/xdist/remote.py#L236-L254); an item without the marker distributes as plain load.

  A plugin can emit the marker but cannot choose the scheduler. So the honest statement is: sharing requires the run to select --dist loadfile, or --dist loadgroup together with an xdist_group marker; under the default --dist load it cannot work by construction, not by implementation defect.

  It is detectable. On the controller, config.option.dist holds the real value at pytest_configure time (verified: serial no, -n 2 load, --dist loadfile loadfile, --dist loadgroup loadgroup, -n auto load). On a worker it is forced to "no" at src/xdist/remote.py#L392-L400, so a guard must gate on the public is_xdist_controller helper.

Cost not previously recorded

Merging has a cost this issue should state, because it strengthens the case for keeping per-block items: one item means one function-scoped fixture setup for the whole namespace. libtmux's own documentation promises the opposite — a fresh server per block — so merging silently changes that contract for every page that shares.

Profiling

Neither shape is a throughput win. A 16-block page, three runs averaged:

• block scope, serial — 16 items, 635 ms
• document scope, serial — 1 item, 662 ms
• block scope, -n auto — 16 items, 3351 ms
• document scope, -n auto — 1 item, 3057 ms

Serially the difference is noise. Under -n auto, worker startup dominates both. The choice is ergonomic in both directions, not a performance trade.

Acceptance criteria

Status is against issue-83-doctest-namespace (#87).

1. A document whose blocks share state passes with sharing enabled, and still collects one item per block. Open — and requires the run to select a compatible scheduler, per Compatibility above.
2. With sharing disabled (the default), a name bound in block 1 is not visible in block 2 — today's behaviour, unchanged. Met.
3. Node ids stay per block and stay machine-independent, satisfying pytest node IDs embed an absolute path #85 without removing the index: page.md::page.md[1], not an absolute path. Met.
4. For blocks that share no state, -k, --deselect, --lf and --junitxml each still address a single block. Met. Scoped deliberately: wherever criterion 1 applies, a block that reads a name an earlier block bound cannot pass alone under any mechanism, loadfile included — so an unscoped version of this criterion is unachievable rather than merely unimplemented.
5. doctest_namespace fixtures are visible in every block under both settings. Met.
6. Sphinx-style groups keep working: blocks in one group share, ungrouped blocks stay isolated, mixed documents behave. Met.
7. A block that raises does not poison later blocks with a half-built namespace, or the limitation is documented. Met and documented: by default a raise stops the namespace so the blocks below never run; --doctest-continue-on-failure runs them against the half-built namespace, producing dependent NameErrors.
8. testsetup / testcleanup run once per group rather than once per block. Met, matching Sphinx's ordering at sphinx/ext/doctest.py#L553 and #L620-L621.

Relationship to the open issues

• Doctest blocks in one document do not share a namespace, unlike pytest text files #83 — the namespace request. This issue is the how: same outcome, without merging items.
• pytest node IDs embed an absolute path #85 — the node-id request. Its Expected has been amended to keep the [N] index, so it no longer contradicts criterion 3.
• Doctest directive options are parsed and then discarded #84 — directive options parsed then discarded. Same construction site, so a fix that re-shapes DocTest creation should land in a known order with this one to avoid two rewrites of the same function.
• Doctest :pyversion: raises InvalidVersion at collection #86 — :pyversion: argument order. Independent bug, same file; listed so the sequence is deliberate.
• doctest: Upstream updates #26 — upstream doctest updates. The clear_globs / test.globs contract used here is the surface that would need re-checking against a new CPython.

Not related: #43 (pytest-asyncio) — that package does not touch doctest collection.

• Share doctest namespaces across a page, and honour directive options #87 — the pull request implementing Doctest blocks in one document do not share a namespace, unlike pytest text files #83/Doctest directive options are parsed and then discarded #84/pytest node IDs embed an absolute path #85/Doctest :pyversion: raises InvalidVersion at collection #86. It takes the merge path, so criteria 2-8 land there and criterion 1 does not.

Alternatives

One DocTest per document. Simplest, and it does satisfy #83 and #85 together. It forfeits criteria 3-4 for the blocks that share — not permanently and not globally: under #87 the default still collects one item per block, and merging happens only where an author declares a group or a project opts into document scope. Sphinx faced the same choice and kept per-block objects.

Group-only sharing, no document scope. Sound, and strictly better for keeping items separate. It does not serve #83's actual reported case — two plain ```python fences — without rewriting them as {doctest} directives, which requires sphinx.ext.doctest in each consuming project's conf.py.

References

• gp-libs doctest_docutils.py per-node loop, v0.0.19 and node naming
• Sphinx group execution: TestGroup, shared ns, test.globs = ns and clear_globs=False, wildcard *, :options: merge, skipped()
• Sphinx doctest docs — group semantics and *
• CPython: DocTest.__init__ copy, DocTestRunner.run, the clear, DebugRunner.run, _extract_future_flags
• pytest: DoctestTextfile.collect, DoctestItem, _check_all_skipped
• pytest-xdist: --dist and its default, loadfile scope split, xdist_group consumption, worker forces dist=no, is_xdist_controller
• myst-parser: a {directive} fence resolves through the docutils directive registry, a plain fence becomes a literal_block whose line is the token's start

[tony]

Agreed on the premise, and #87 already keeps per-block ids in the case that matters most — but the proposed mechanism does not survive -n auto, which is the constraint the issue does not mention and which decided the design.

What #87 does today

Sharing is not the default. An ungrouped page collects one item per block, exactly as before:

$ pytest page.md --collect-only -q

page.md::page.md[0]
page.md::page.md[1]
...
16 tests collected

So criteria 2, 3 and 5 hold as written, and criterion 3's reconciliation of #85 is the reading #87 took: the [N] index stays, only the absolute path is gone. pytest 'page.md::page.md[6]' selects and passes. Criteria 6 and 8 hold. Merging happens only where an author declares a group, or where a project opts into --doctest-docutils-namespace-scope=document.

That leaves one real disagreement: criterion 1, which asks for sharing and one item per block.

Why not test.globs = ns with clear_globs=False

I built that version first. It works serially and fails under distribution, because pytest-xdist runs items in separate worker processes and a shared mapping does not cross them. On the two-block page from #83:

$ pytest state.md -q

2 passed

$ pytest state.md -q -n 2

1 failed

with NameError: name 'greeting' is not defined. The merged mechanism passes both. This matters concretely: libtmux runs uv run py.test ... -n auto in CI with docs/ and README.md in testpaths, so a shared page there would be split across workers.

Criterion 4 under the proposal

The ids survive, but addressing a block does not run it. With the shared-mapping build, on the same page:

$ pytest 'state.md::state.md[1]' -q

1 failed

and --deselect of block 0 gives 1 failed, 1 deselected. So -k, --lf and --deselect reach the block but cannot execute it once it depends on a name an earlier block bound. That is inherent to shared mutable state across independently-runnable units rather than a flaw in the proposal — but it means criterion 4 is nominal rather than real whenever criterion 1 is in force. Merging resolves it by making the addressable unit the namespace, so whatever you can select, you can run.

On the benchmark

The table matches what I measured, and it is worth keeping in the issue: per-block items are not a throughput win, so the case for them is debuggability. Where sharing is off — the default, and every downstream page today — #87 keeps that debuggability in full.

What is genuinely lost

Inside a declared group, or under document scope, the group's blocks become one id. A block gated end to end is lifted back out and keeps its own id, but the rest do not. If per-block addressability inside a shared group is worth more than surviving -n auto, that is a real trade and I would rather it be decided explicitly than inherited from my implementation. The alternative that keeps both is to leave sharing scoped to groups and never offer document — narrower than #83 asked for, but it confines the loss to blocks an author deliberately joined.

Measurements above are reproducible against issue-83-doctest-namespace; the shared-mapping build is kept as a stash if it is useful to compare directly.

[tony]

Criterion 7 is now satisfied explicitly rather than by implication.

Measured on a two-block group whose first block raises:

$ pytest raise.rst -q

1 failed

The blocks below never run, so nothing sees a half-built namespace — stopping is what protects them. The exception is the flag that asks for the opposite:

$ pytest raise.rst -q --doctest-continue-on-failure

UNEXPECTED EXCEPTION: ZeroDivisionError('division by zero')
UNEXPECTED EXCEPTION: NameError("name 'value' is not defined")

There the later block does run against a namespace missing what the failed example would have bound, and one broken line reports as a first failure trailed by dependent NameErrors. That is now stated in the how-to's costs list rather than left to be discovered.

Status against the acceptance criteria, on issue-83-doctest-namespace:

• 1 — not met as written, and it is the one open disagreement: sharing collapses a group to one item. The measurements in my previous comment are why.
• 2, 3, 5 — met. Default is one namespace per block, ids keep the [N] index and carry no absolute path, doctest_namespace fixtures reach every block under both settings.
• 4 — met at the default scope. Under sharing it is met in the sense that ids exist and select, but a selected block cannot run alone, which is inherent to shared state rather than to either mechanism.
• 6 — met: grouped blocks share, ungrouped blocks stay isolated, and mixed pages behave.
• 7 — met, now documented.
• 8 — met: testsetup and testcleanup run once per group, before and after the rest of it, whatever order the page writes them in.

Criterion 1 stays open on purpose. I would rather it be closed by a decision than by my implementation.

[tony]

Your --dist loadfile finding stands, and it corrects me. I stated the constraint as "sharing does not survive -n auto", which is wrong as written: it is specific to the default load scheduler. Reproduced here against the shared-mapping build:

• serial — 2 passed
• -n 2 (default load) — 1 failed, 1 passed
• -n 2 --dist loadfile — 2 passed
• -n 2 --dist loadgroup, no marker — 1 failed, 1 passed

The last line is the one that carries the weight, and it is the half neither issue records: loadgroup only helps once something emits xdist_group, and a plugin can emit the marker but cannot choose the scheduler. So the accurate statement is not "shared namespaces break under xdist" but:

Shared namespaces require the run to select --dist loadfile, or --dist loadgroup together with an xdist_group marker. Under the default --dist load they cannot work, by construction rather than by implementation defect.

That belongs in this issue's Compatibility section as a fifth axis.

Criterion 4 should also be scoped. As written it promises that -k, --deselect, --lf and --junitxml each address a single block, which cannot hold wherever criterion 1 applies: a block that reads a name an earlier block bound cannot pass alone, under any mechanism, loadfile included. Suggested rewording: "For blocks that share no state, each of -k, --deselect, --lf and --junitxml addresses a single block."

Two measurements that change what the remaining middle is worth. page.rst:11 in a merged failure now points at the failing block's own line — that was not reliably true until the phase-ordering fix landed today, where a page whose testsetup sat below its group reported every example against the wrong line. And doctest already caps the %03d gutter: a failure in the last of fifteen merged blocks prints ten lines, not the whole page.

So naming the group and block index in the failure header — which I proposed and you called the best idea in the exchange — turns out to be mostly redundant. Finding the block is already served by an accurate file:line, and the remedy for it not being accurate was a bug fix, not a new reporting channel. I would rather say that than build a DoctestItem subclass to re-deliver it.

That leaves the shape you proposed: merging as the default, and per-block sharing available for projects that pin loadfile. That is a third mechanism rather than a variation on the current one, so it wants its own decision and its own issue rather than being folded in here.

#85 is amended — its Expected now keeps the [N] index, matching criterion 3.

[tony]

Superseded by #89, which carries this issue's evidence and turns its one remaining ask into a concrete proposal.

Of the eight things asked here, seven are done in #87: isolation stays the default, node ids keep the [N] index and drop the absolute path, doctest_namespace fixtures reach every block, groups behave, a raise no longer leaves later blocks half-built, and testsetup / testcleanup run once per group. The one asking that -k / --deselect / --lf / --junitxml each reach a single block holds wherever blocks do not share state, and cannot hold where they do — under any mechanism, --dist loadfile included.

That leaves one open question — sharing state while each block stays its own test — and #89 now holds it, along with the evidence from here: what per-block ids buy day to day, the benchmark showing they are not a throughput win, and the fixture-lifetime cost that libtmux's own contributor docs make explicit.

The correction this issue produced is recorded there too: the distribution axis, which was missing from the compatibility analysis and which governs the answer. Thanks for pushing on it — --dist loadfile is what makes the proposal shippable rather than merely arguable.