TS engine extensions (epic integration branch) - #416
Merged
Conversation
…pi foundation, TsEngineHost, EngineRegistry Arc migration Restructure the engine-extension epic: split Plan 1a into protocol/host/engine sub-plans, extract Plan 2A, migrate the EngineRegistry definition into plan1a-engine, and rename `quarto build-ts-extension` -> `q2 build-ts-extension`. Engine-A foundation: - Publish @quarto/api + @quarto/types skeletons: config key-lists, faithful metadata partition, and the platform/text/format/crypto/ mappedString/markdownRegex/console/path/system namespaces. - Add the TS engine-host JSON protocol types; remove the vestigial FormatIdentifier::Custom variant. - Land quarto_runtime_dir(), ExecutionError::Timeout/ProcessCrashed, and the TsEngineHost transport split + demux. - plan1a-engine Phases 3/3.5/4: trait-discovery surface, resolve_engines + AST language scan + enforcement, and the EngineRegistry -> Arc migration + two-step TsEngine lifecycle. (This is the Arc/field-type half of the pipeline.rs merge conflict.)
…ion + RTQ wire surfaces Q1 engine-lifecycle ground-truth research; plan1a/1b reconciliation (embedding model, concurrency, return-based deps); the RTQ course-correction split (host-bugs plan + Plan 5 pooling stub); the engine surface-coverage audit and design questions. Code (RTQ items): - item-A: split EngineHostContext into Init/HostGlobalConfig + LaunchEngine/EngineProjectContext. - eng-1: complete the discovery tier (generates_figures, can_freeze, quarto_required on LoadEngineResult). - fc-1: carry metadata/pandoc/resourceFiles/preserve + postProcess on the execute-result wire. - fc-2: deferred-dependencies wire surface (Dependencies verb).
…ineExecutionStage + resolve_engines + EngineClaimsFileStage The engine-host-deno package: newline-framed JSON transport, MappedString rehydration, metadata-as-format partition, the Deno PlatformHost, buildQuartoAPI + engine-loader, the run-loop, execute-dispatch, a per-engine execute queue with cooperative Cancel + poison/transparent relaunch, and the committed esbuild bundle + xtask. MappedString unification into @quarto/types (Plan 1b.1). Plan 1c Tasks 1-14: - engine-contribution types + static-claim conversion; parse contributes.engines with validation/warnings. - TsEngineHost observability; TsEngine static claims + execute-time validation; quarto_data_dir / is_interactive / running_in_ci. - registry contribution_order + shutdown_all; build the real engine registry on ProjectContext. - Task 8: thread the project registry onto StageContext; make EngineExecutionStage stateless. - Task 9: resolve_engines drives execution (claimed short-circuit, contribution_order); delete KNOWN_ENGINES. - Task 10: EngineClaimsFileStage + LoadedSource.conversion + C'. - Task 11: P2-12/P2-13 failure model; Task 12: q2 build-ts-extension CLI; Tasks 13/14: echo/echo-legacy fixtures + E2E render gate. (This is the structural half of the pipeline.rs conflict: builder-signature removal, engine-stage reconstruction, EngineClaimsFileStage insertion, pipeline-length bump.)
… Julia & marimo engine validation, preview-capture fixes Plan 2 (Phase A/B): deferred launch-context bodies + global seam, execProcess restore (mergeOutput/stderrFilter/timeout; stdin is a mode), LanguageClaim type + claim constructors, and derived QuartoAPI conformance (retire the harness cast). Plan 3: full @quarto/api/jupyter port (13 tasks) — MIME dispatch, tags, labels, cell-options, preserve, widgets, to-markdown, percent-script, assets, makeJupyter; wire makeJupyter into buildQuartoAPI (cast-free). Decision: rename claims-files -> claims-extensions. Plan 4 (Julia, Phases 4A-4J): minimal render, figures, multi-cell/error handling, engine-file directory expansion, symlink-cycle hardening, frozen test seams; plus preview validation. Plan 4c (marimo, Phases 4cA-4cH): fixture, minimal render, sql-interop, the bare-sql ownership-gate fix (handledLanguages is leave-alone), and SC8-SC21 seam tests. Preview-capture (bd-h4rhohhy): Bug A/B/C repro + fixes — realistic .cell wrappers, bound splice, reader stray-line escalation bound, error-path coverage. Widen static engine claims to Vec-per-language.
…aims-extensions Plan 9 (q2 call engine) and Plan 10 (checkInstallation / q2 check): Q1 spec extraction, implementation research, and frozen-test-seam plans. The engine:/engines: two-key grammar design contract. Plan 6 (Pass-1 engine resolution) plan build + review rounds. Fix registry BUILTIN_ORDER to match the resolver. Code (plan1c.2 P2/P4): - typed FileClaim + parse-time extension normalization + to_wire_ext. - RenderableExtensions seam + claimed_file_extensions; discover statically-claimed engine extensions in projects. - pub(crate) contribution_order getter; real-process crash-path E2E (ProcessCrashed with stderr).
…ests, marimo preview splice Plan 4b/4d: case-4 becomes engine self-enforcement; owned_languages positive-ownership wire field. Plan 4b Phases A-F: - synthetic resolution-shaped + behave fixtures; resolution-tier matrix (pure-unit rows + real-Deno rows). - splice project _quarto.yml engines: into ordering (Phase C). - jupyter ANSI-strip binding + widget E2E (Phase E). - cooperative-cancel poison, crash-triggered relaunch (resettable transport, generation guard), Dependencies-verb round-trip (Phase F). Marimo q2 preview capture-splice fix (splice unwrapped engine islands, bd-5jxcio5d) + engine include-in-header delivery so widgets hydrate (bd-5oyk1xce). Promote plan1a.6 (off-stdout -> loopback TCP) to a plan of record. Relax over-required _extension.yml fields (title/author) to match Q1.
…ofile
Plan 6 Phases 1-5:
- claim-schema list shorthand + pub(crate) claim parsers.
- resolve_engines_inner consults generated-languages; whole-table claim
replacement; load-free predicate + no-load claim surface.
- Stamp Pass-1 engine resolution onto DocumentProfile (the
engine_resolution field + DOCUMENT_PROFILE_VERSION bump).
- Thread _extension.yml provenance into the engine-extension cache key;
add the Pass-1 counter + fall-through warning.
- Accept bare kind-string claim map values ({r: primary}); user-facing
docs for the engine:/engines: keys.
(This bucket owns the document_profile.rs version-bump conflict.)
…d-ts-extension' + fixture cleanup plan1a.6 (off-stdout -> loopback TCP), Phases 1-4: - TcpTransport newline-JSON framing, accept_and_handshake, spawn_into_tcp, shutdown + crash-reap, large-payload deadlock-freedom. - Deno dial-back: connectControl + main.ts selection. - Flip the production engine-host to loopback TCP; malformed control-socket frame is fatal. - Delete the stdio transport; retire the "stdout-is-the-protocol" / console.log-corrupts contract. plan1c3: rename `q2 build-ts-extension` -> `q2 call build-ts-extension`, extract the extension-bundle build lib (CLI becomes a wrapper), and hermetic self-regenerating synth fixtures (regenerate at test time). Capture-splice fixes: keep the earlier engine's cell output in multi-engine folds (bd-5oyk1xce Bug B); splice an echo:true cell's full output run. Julia fixture cleanup: drop Plots.jl, remove the orphan plot.qmd trap, surface swallowed render errors.
… main Mechanical compile/test fixups after rebasing the 8 engine commits onto main. These live in main-authored files that no branch commit owns, so they can only be applied as a post-rebase sweep (kept separate to preserve the 8 rebased commits intact): - Arc<EngineRegistry> migration call-sites main added since the merge-base: quarto-hub-provider Provider.registry field + new() param + 3 test call-sites; preview_record test. - EngineCapture.files (#410) on branch-side literals in capture_splice tests. - ProjectContext new fields (binary_dependencies/extensions/registry/…): ..Default::default() on test literals (pipeline, date_normalize, mermaid, language_pipeline, printable_render, video_shortcode_preview). - CodeBlock.code_start (#412) in the jupyter text_execute test helper. - Adapt main's #410 FigureWritingTestEngine to the branch's claims-based engine-resolution model (implement claims_language) so capture_embeds_engine_supporting_file_bytes exercises the engine again. - Drop stale `pub mod ts_process_framing_probe;` (file deleted by plan1c3 fixture cleanup; kept title_block_pipeline + video_shortcode_preview, which are main modules whose files exist). - Regenerated: package-lock.json (npm install), .braid/snapshot.jsonl.
…tone Plan 8 Grand plan (2026-04-16-ts-engine-extensions-subprocess.md): - Mark Plans 4b, 6, 1a.6 complete; add missing Plan 1c3 row (complete). - Tombstone Plan 8 row; refresh totals + status-key legend (add tombstoned). Plan 8: TOMBSTONED — diagrams are not engines. Mermaid is now an AST transform on main (transforms/mermaid.rs), HANDLED_LANGUAGES is gone, and 2026-07-20-mermaid-regular-rendering.md (bd-5m4ga0s1) pivots mermaid off the engine model. Both Part A/B depended on that premise. Banner + status added; original content preserved. Plan 6: check off the user-facing-docs box — the docs landed in the Plan 6 commit itself (docs/guides/authoring/engines.qmd) but the box was missed. Plan 5: add a Plan 1a.6 (loopback-TCP) interaction note (vindicates the EngineTransport seam; re-target the measure-first gate to include the TCP handshake; crash path must use TCP liveness not socket-EOF; kernel-survival premise unchanged) and fix the stale StdioTransport -> TcpTransport reference.
Contributor
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
340 commits behind main since 2026-07-24. Records the measured basis for choosing merge over rebase (naive rebase stops at 7 of 10 commits and re-resolves the same hub files 2-4x each), the 23-file / 50-hunk conflict inventory in resolution order, and the one genuine re-port (project/discovery.rs: RenderableExtensions onto main's rewritten module). Points CURRENT.md at the new runbook.
Restructures the ts-engine-extensions <- main runbook into three phases: Phase A resolves the 23 merge conflicts, Phase B lands the five design changes the merge forces (each its own commit so the merge stays reviewable), Phase C verifies. Records seven decisions settled with Gordon on 2026-08-14/16, each with rationale so an implementer needn't re-derive them: - D1: engine-claimed extensions widen the default render pattern set (overrides main's static-list decision from bd-6d2wj4zp, 2026-08-07) - D2: only static claims feed discovery; dynamic claimers fall through silently, matching the existing Pass-1 language-claim precedent - D3: SourceType drops Ipynb/Rmd; the field becomes Option<SourceType> - D4: EngineClaimsFileStage -> SourceConversionStage; claims_file stays (wire protocol + public engine-author API) - D5: refuse engine claims on the whole native set with new Q-2-50; this also removes the need to touch main's Q-2-40 guard - D6: DOCUMENT_PROFILE_VERSION -> 11, and stop bumping it on the branch - D7: conversion provenance (bd-zlemoc6w) is before-PR, not this merge Also records why rerere is explicitly disabled: the repo's rr-cache holds June/July resolutions that replay stale, reverting main's newer work in at least four verified places.
Consolidates two review streams (my own re-read, and a blank-slate Opus agent asked to review as an implementer) plus git archaeology on build_wasm_html_pipeline. 21 findings; 20 applied here. Two were serious: - A5 said "make has_renderable_extension consult the set (the set is a superset)". FIXED_RENDERABLE is &["qmd"] — md is NOT in it — so a literal implementation would drop .md from discovery entirely and silently revert bd-6d2wj4zp. Now specifies FIXED_RENDERABLE = ["qmd","md"] (or an explicit disjunction) and calls out the trap. - B1's WASM guard rested on a false premise. The stage's own doc comment claims "native pipeline builders only"; that comment is stale. build_html_pipeline_stages_with_options is not cfg-gated and the live WASM entry points reach it, so WASM already converts and already hard-errors on unclaimed extensions. The guard item is removed and replaced by B1b: delete build_wasm_html_pipeline, which archaeology shows has never had a production caller since Carlos added it in 094c62a (2026-01-28), was born redundant (its own plan marks the WASM entry-point phase already done), and silently drifted when Task 10 (d66f4f0) updated the shared builder without it. Also fixes the stale comment that caused the error. Other substantive fixes: D1 now specifies how RenderableExtensions distinguishes engine-claimed from fixed members (it is a flat HashSet today and cannot); the A5 widening must exclude the native set unconditionally rather than depend on B3 landing; D3 pins the Option<SourceType> terminal state for pass-through files; A8 gains the silently-colliding pipeline.rs stage-count assertions (23 vs 24, file auto-merges) with "observe, don't hand-count" guidance; Phase C gains deno as a stated prerequisite and a zero-SKIP criterion, since the engine e2e suites skip silently and the suite can be green with every test binding D1/D2/D5 skipped. Corrections: discover.rs has 7 hunks not 6; parse_config has 2 call sites not 5; render.rs has no RenderableExtensions construction site; relative_to_dir added to A5's delete list with its known callers; lockfiles auto-merge; Q-2-50 emits once per file; bd-zlemoc6w rephrased as a post-runbook gate rather than an unsatisfiable checkbox; stale state table refreshed.
Phase A of claude-notes/plans/2026-08-13-ts-engine-extensions-merge-main.md. Resolution philosophy throughout: keep the branch's structure AND main's new features; never take a whole side of a hub file. Re-measured against main at f8df952 (the runbook was written against 0dcd7e8, since when main gained ~100 commits / 261 files): - 26 conflicts, not the documented 23. The three extra are `dependency.rs`, `xtask/main.rs`, `xtask/dev_setup.rs`. - DOCUMENT_PROFILE_VERSION is 12, not 11: main independently reached 11 (bd-toc-smart-quotes), and D6's rule is "main's current value + 1". - project/discovery.rs, extension/types.rs, stage/data.rs and listing_generate.rs were untouched by the delta, so A3/A5/B1 were unaffected. Conflict resolutions of note: * discovery.rs (A5) re-ported onto main's rewritten module rather than 3-way merged. Branch's hand-rolled glob code deleted in favour of main's crate::glob; FIXED_RENDERABLE is {qmd, md} so `.md` cannot drop out of discovery; RenderableExtensions gained a second field holding the engine-claimed list (D1 needs "which members came from an engine?", and the flat set cannot answer it). * D1 implemented in effective_render_patterns, not the diagnostics path: with no positive user pattern the effective set is `**/*.qmd` plus one `**/*.<ext>` per engine-claimed extension, each carrying SourceInfo::generated(By::programmatic_config()), and excluding the native set unconditionally so `.md` stays opt-in (bd-6d2wj4zp survives). render_pattern_diagnostics still sees only the user's patterns, so synthetic globs cannot raise a spurious Q-5-13. * dependency.rs (not in the runbook's inventory): main added dependency versioning, the branch added a name-collision first-wins guard. The guard now keys on the *version-qualified* prefix — otherwise two versions of one dependency read as a collision and main's two_versions_of_one_dependency_produce_two_artifacts would warn and drop the second. * Q-2-40 (.md never executes) reads the *declared* engines, not the resolved ones. On this branch resolve_engines derives its sequence from the languages present, so a `.md` with `engine: jupyter` and no code cells resolves to an empty sequence; the warning is about the spec the author wrote. Factored explicitly_declared_engines() out of prepare_resolution so the guard and the resolver share one definition (it covers both `engine:` and the top-level shorthand, resolved against registry names rather than main's hardcoded KNOWN_ENGINES). * Extension intake diagnostics are surfaced, not dropped: discover_extensions gained a diagnostics return on main, and the branch's pre-walk discover_extensions_only half had no sink. They now fold into ProjectConfig::config_diagnostics. * jupyter text_execute: main's kernel_scope guard sits below *both* early returns, so an all-cells-ceded passthrough never acquires a kernel. Test adaptations where main's tests met branch design decisions: - engine_blocks_attribute_to_intermediate_after_config_registration needed a real `{stub}` cell: a cell-less document now short-circuits to a markdown passthrough with an empty sequence, so an `engine:` key alone no longer triggers execution. Also moved off main's with_registry() constructor, which the branch removed when the stage became stateless (registry comes from ctx.registry). - echo_engine_e2e projects TocEntry::title through inlines_to_string (profile v11 made outline titles Inlines). - read.rs keeps the branch's author_optional / q1_engine_shape Q1-compat tests (absent on main), adapted to Option<String>. Snapshots: 6 files (3 added, 3 modified), ALL byte-identical to main's versions — purely incoming, none regenerated here. Added: entity-references, smart-typography-positions, llms_companion_rich_content. Modified: the three listing_pipeline categories snapshots. Verification: cargo build --workspace clean; cargo build --workspace --tests clean; cargo nextest run --workspace = 12615/12616 passing. The one failure, quarto-preview config_reports_embedded_asset_manifest_hashes, is a stale build artifact, not a resolution defect: q2-preview-spa/dist/ in this worktree dates from Jul 24 and so predates main's addition of scripts/manifest-dist.mjs to the SPA build, leaving no spa-manifest.json for the test to find. Rebuilding the SPA (Phase C's full xtask verify) regenerates it. Phases B and C of the runbook are not in this commit.
…Type q2 processes exactly two kinds of markdown: qmd (with code execution) and md (without). Everything else is converted to qmd *before* the parser, by a stage that asks the installed engines what they claim. A static, engine-specific format list stops making sense once engines arrive as extensions, so this replaces the enum dispatch with dynamic dispatch at the same insertion point the January 2026 design (806703b) chose. SourceType loses Ipynb/Rmd; the field becomes Option ---------------------------------------------------- `Ipynb`/`Rmd` never acquired a behavioral consumer — the intended one was a `PipelinePlanner` pushing a `ConvertNotebook` stage, never built. Under this model bd-19nc56ao (`.ipynb`) becomes an engine claiming `.ipynb`, and `.Rmd` becomes knitr claiming `.Rmd`. With only two variants the old `unwrap_or(SourceType::Markdown)` stopped being a shrug about unknown formats and started asserting "plain markdown, never executes" about a file about to be converted and executed. Contract: - `LoadedSource::new` stores `from_path()` directly, no fallback. - `None` = unknown extension, not yet converted. - The conversion stage stamps `Some(Qmd)` on the conversion branch only; the pass-through branch is already correct from load. - Invariant: after that stage `source_type` is always `Some`, because an unclaimed non-native file hard-errors and never gets further. Trace JSON now renders the variant (`"Qmd"`) or `null`, not the Option's Debug form (`"Some(Qmd)"`). No compatibility concern: SourceType is not serde-derived and reached trace JSON only via `{:?}`. EngineClaimsFileStage -> SourceConversionStage ---------------------------------------------- EngineClaimsFileStage -> SourceConversionStage "engine-claims-file" -> "source-conversion" engine_claims_file.rs -> source_conversion.rs Conversion is the action; claiming is only the predicate that selects it. The new name stays accurate if built-in (non-engine) converters appear later, it reads correctly immediately before `parse-document`, and it follows the house convention (`metadata-merge`, `language-resolve`, `include-expansion`). `claims_file` / `claimsFile` / `ClaimsFile` are deliberately NOT renamed. That name is not internal: it is a wire message pair (ToEngine::ClaimsFile / FromEngine::ClaimsFileResult) and a required export of the public engine-author API — engine-loader.ts throws "engine module … is missing required export: claimsFile" if an extension omits it. Renaming it would break every third-party engine. The engine claims; the stage converts. That rationale is recorded in the module doc so the asymmetry doesn't read as an oversight. Refuse engine claims on the native set, with Q-2-50 --------------------------------------------------- An engine that claims `.qmd`, `.md`, `.markdown`, or an extension-less input would sit in front of Quarto's own parser and decide what the markdown means. Refuse the claim, warn once, fall through to the normal pass-through path. Scope is the whole native set, not just `.md`: both markdown kinds are q2's own. This closes a real bug, not just a policy gap. Without the guard the claims loop is ungated, so an engine *can* claim `.md`; the file would then be converted (source_type stamped Qmd) and have execution suppressed anyway, because the Q-2-40 guard reads `SourceType::from_path` of the *original* path — still `.md` — and emits a spurious "engine specification ignored" warning for a conversion that silently happened. Neither side's tests caught this: main has no claiming engines, this branch had no guard. Consequence: Q-2-40 needs no change, and engine_execution.rs is untouched. The diagnostic is emitted once per FILE, naming every refused engine — not once per claiming engine. The refusal sits inside `for engine in &engines`, so a naive `continue` with an inline warning would produce N diagnostics for one file when N engines claim it; a test pins exactly that with three greedy engines. The native set is read from `discovery::NATIVE_EXTENSIONS` rather than re-spelled, so this guard and the discovery exclusion cannot drift apart. Ships the `Q-2-50` catalog entry and `docs/errors/markdown/Q-2-50.qmd` in the same commit, as error-docs-page-missing requires. The page is written for the extension *author* — the person who can fix it — and points at `claimsLanguage` for the case where a language claim was what was wanted. Delete build_wasm_html_pipeline ------------------------------- Dead code, not a seam: zero production callers — nothing in crates/wasm-quarto-hub-client/ or hub-client/ ever referenced it. Callers were two integration tests, one unit test, and a re-export. Its own originating plan (2026-01-28-unify-hub-client-pipeline.md) marks "Update WASM Entry Points" as already done — the browser path already went through the shared builder. It was born redundant and had silently drifted: d66f4f0 added the conversion stage to the shared builder and did not touch this one, with no test catching it. No unique coverage lost: both integration call sites asserted invariants already asserted against `build_html_pipeline_stages` a few lines away, so the retargeted assertions would have been duplicates and are dropped. Also fixes the stale doc comment claiming the conversion stage is inserted "in the **native** pipeline builders only" — false, and the source of a false premise in the merge runbook's first draft. Three other doc comments naming the deleted builder (math_js, feed/mod, feed/link_inject) now name the shared builder. Discovery coverage for the engine-claimed path ---------------------------------------------- Every discovery test in the tree used `RenderableExtensions::fixed()`, so the entire engine-claimed path had NO coverage on either side of the merge. Seven tests close that: t6 — walk path: a claimed `.echo` is admitted, and the SAME exclusions that govern `.qmd` (underscore, dot, output-dir) still apply to it, while a non-member `.ipynb` stays out. Binds that engine extensions flow through the same predicate, not a bypass branch. t6b — pattern path: an explicit `render: ["*.echo"]` selects it too. t7 — membership is the gate: naming `*.ipynb` in `render:` does NOT make an unclaimed extension an input. d1_claimed_extension_renders_with_no_render_key — a project with a claimed `.echo` and no `project.render` key renders its `.echo` files, still renders `.qmd`, and still does NOT render `.md`. d1_widening_emits_one_generated_glob_per_claimed_extension — exactly one synthetic glob per non-native claimed extension, each with generated provenance; `md`/`qmd` never widen even when claimed. d1_widening_does_not_apply_when_a_positive_pattern_exists — a user-written positive pattern is authoritative, no widening. d2_dynamic_claimer_contributes_no_wildcard_and_no_warning — a `claims_files: None` engine declares no static extension, so it contributes no member and no glob, and the fall-through is SILENT. The module doc still described discovery as `.qmd` and `.md` only; it is refreshed. (A later commit supersedes the default-pattern widening these tests cover; the walk-path and membership bindings survive it.) Verification: cargo build --workspace --tests clean; cargo xtask lint all checks passed (1018 files); cargo nextest run -p quarto-core = 3921 passed, 0 failed.
…unbook
Docs only; no code, fixtures, or snapshots touched.
7-series: a native, engine-agnostic content-processor registry
-------------------------------------------------------------
Reorganizes Plan 7 into a series, fixing the Pass-1 perf bug where
admitting a TS-engine percent/spin script launched Deno (or Rscript for
knitr spin) in the indexing pass via wire markdown_for_file.
- 7b (NEW, solid): percent + spin processors run natively in Rust, named by
engines via `processor:` on claims-files. Sniff + convert + A+ SourceInfo
in-process => zero Pass-1 launch for built-in and TS engines alike.
percent ports percent-script.ts; spin reimplements knitr::spin with
matchable via the already-present tree-sitter-r (native + wasm32, zero
extra WASM cost). 10-phase TDD checklist + 13-row test-seam spec +
Plan-6/4b coordination.
- 7 (root): reframed as the content-processor architecture umbrella.
- 7a (tombstone): arbitrary-regex content-pattern withdrawn; survivors
(discovery admission, coherence, built-ins-as-data, Q6 membership-cache
contract) migrated into 7b.
- 7c: promoted from a 46-line placeholder to a full ipynb plan. Three ipynb
designs existed (7c stub, the 2026-07-20 surface-syntax design, merge
runbook D3); 7c is now the single home. It absorbs the surface-syntax
body — the non-affine JSON argument, per-cell ephemeral SourceFiles as
the coordinate root, Q1-compatible cell emission, presentation P1/P2,
stored-output replay (option B), the three implementation seams — and
records D3 as the settled attachment point (`.ipynb` = an engine claiming
`.ipynb`, converted by SourceConversionStage). The stub's open
native-vs-wire question is answered: native, by the series'
zero-Pass-1-launch invariant.
7c drops two inherited assumptions the July-2026 audit falsified, both of
which REMOVE work: no `SourceInfo::NotebookCell` variant (cell identity is
per-file, not per-span, so the closed enum with ~8 match sites is
untouched), and no sidecar (run_pipeline is in-memory; no qmd intermediate
is written).
7b gains the forward-compatibility obligation this surfaced: `Converted`
needs a channel for ephemeral source files (`files: Vec<(String, String)>`).
Percent and spin never surface it; ipynb's pieces point at virtual per-cell
files, so without it 7c would have to change convert()'s return type —
exactly what those obligations exist to prevent. The sidecar-envelope
obligation is suspended pending a named consumer.
The 2026-07-20 doc is retitled a design reference pointing at 7c. Its
design stands; only its attachment point (SourceType::Ipynb keying,
registry declined as speculative) is superseded. Grand-plan 7-series rows
reconciled so the index stops contradicting the tombstone it now carries.
Merge runbook: reconcile the checklist with what happened
--------------------------------------------------------
B4's third item ("If done: .envs(project_env.iter()) at the deno spawn…")
is now UNCHECKED and struck through, because it was deliberately not done.
The A and B items had been blanket-checked in one pass without individual
verification — exactly the failure the finishing-a-plan rule warns about.
The sizing result and off-ramp decision are recorded inline, with the
reason the naive fix is wrong rather than merely large: the deno host is
spawned once and shared across every document, so env applied at spawn time
would be whichever document started it first. Strand: bd-x30l7ee5.
Phase C evidence recorded with real invocations and inspected output for
the claimed-extension discovery case and for the Q-2-50 native-set refusal
(the file passes through unconverted, exactly one warning per file), plus
the zero-deno-skips check that makes those results meaningful.
Full `cargo xtask verify` (not --skip-hub-build): all 14 steps passed;
step 5 alone is 12,624 tests passed / 0 failed. `q2 render docs/` renders
244/245 with 258 pages inspected; the 1 error is a missing jupyter runtime
on this machine, not a regression.
Two local tool gaps closed on the way, neither a code defect: absent
`wasm-opt` (binaryen), a requirement main newly introduced in 8024198
and that `cargo xtask dev-setup` installs; and a stale
`q2-preview-spa/dist/` predating main's `manifest-dist.mjs`. `cargo xtask
verify` cannot self-heal the latter — Rust tests are step 5, the SPA
rebuild is step 7.
Also records a worktree gotcha: `stage-doc-examples` writes to the main
checkout rather than the worktree it runs in, silently, because
repo_root() resolves --git-common-dir. Filed as bd-afi4avsf.
An engine's only diagnostic channel is `registry.diagnostics`, and that vec was written and never read — the missing-static-claims warning was built, pushed, and discarded. Verified pre-fix by rendering a project with the legacy-python fixture: nothing printed, exit 0. Making claim failures loud is pointless while that channel is a black hole, so both halves land together. Drain registry.diagnostics -------------------------- The sink already existed. `ProjectRenderSummary::project_diagnostics` is already counted by `diagnostic_counts`, promoted by `promote_warnings_to_errors`, and printed in both the text and --json-errors paths (and covers single-file renders, which also go through ProjectPipeline). So this drains into it rather than building a second channel. Drained at END of render, not after registry build: engines push into the same Arc *during* Pass 1 / Pass 2, so an init-time drain would miss Q-16-12 entirely. Also drained at the --fail-fast early return, which builds its own summary and would otherwise swallow everything. Q-16-11 (pass-1 fall-through) moves off a bare `eprintln!` into the same vec. The print is equivalent — the project_diagnostics loop is outside the `quiet` gate, as the eprintln was — but it is now counted and --strict-promotable (bd-yjs54ptg / GH #220), which a raw stderr write can never be. It prints at end of render rather than between the passes. Q-16-10 is rebuilt with DiagnosticMessageBuilder: it was a ~380-char single-line title, now a structured block, and it points at `_extension.yml` (the file a user edits) instead of the `.js` bundle (a build artifact). It is deliberately NOT gated on the engine being used by the render. Stop swallowing claim-load failures ----------------------------------- `claims_language` and `claims_file` mapped a failed load to "no claim" and discarded the error. Effect: a page with the user's code echoed instead of executed, exit 0, nothing anywhere saying the engine failed; or a misleading "Can't determine execution engine for <file>" blaming configuration. Neither Err arm cached, so a broken engine was re-loaded on every claim probe of every document — a failing subprocess load per probe. A sticky `load_failed: OnceLock` fixes both: setting it emits Q-16-12 exactly once, and checking it short-circuits the dynamic branches before ensure_loaded. Callers still answer "no claim". Resolution must stay infallible, or one broken extension prevents a project rendering at all, including documents that never touch it. Terminology: these are Q1 dynamically-claiming engines that have not been updated with static claiming — not "undeclared" engines, which reads as the engine being missing entirely. They work; they just cost a subprocess load. Never load an engine to ask about a file type q2 owns ----------------------------------------------------- q2 refuses an engine claim on the native set (`""`, `qmd`, `md`, `markdown`) outright — that is Q-2-50. But `SourceConversionStage` asked anyway: it called `claims_file` first and applied the refusal second. For a Q1-style dynamically-claiming engine, `claims_file` answers by loading, so a project containing nothing but `.qmd` spawned a deno subprocess per engine to produce an answer the very next line discarded. The native-set check is hoisted above the claims loop and decided from static declarations via a new `ExecutionEngine::try_claims_file` — the file counterpart of the existing `try_claims_language`, side-effect-free in the same way (no load, no cache write, no static_file_answers record). `try_claims_file` defaults to `Some(false)`, NOT `None` like `try_claims_language`. Different reasoning: there, `None` is fail-safe because a missed language claim silently misroutes a cell. Here the default mirrors `claims_file`'s own `false` — no built-in overrides `claims_file`, so "definitively does not claim, no load needed" is simply true for every engine that has not opted in, and `None` would falsely assert that built-ins might need loading. The trade: Q-2-50 can now name only STATIC claimers of the native set. A dynamic claimer is not asked, so it cannot be reported — we decline to obtain the answer. The diagnostic is kept where it is free (a declared `claims-files: [".md"]` is the case actually worth telling an extension author about) and dropped where it costs a subprocess. That also settles Q-16-10's wording. "must start it as a subprocess on every render" was true of the drain-and-report step and is false once the native-set check is static — a markdown-only project loads nothing. Fixed in all three places it appeared: the Rust message, the catalog message_template, and the docs page, whose "Why this warns even when the engine is unused" section argued from the now-wrong premise and now explains the real position with a pointer to Q-2-50. Tests ----- - 9 e2e tests driving the real q2 binary. - `markdown_only_project_loads_no_engine` binds that a markdown-only project loads nothing. Its predecessor (`markdown_only_project_still_loads_the_engine`) bound the behavior the static hoist removes and is deleted rather than kept: a test that argues against the design is worse than no test. - MockEngine gains `try_claims_file` mirroring its `claims_file`. Its `claimed_extensions` is a fixed list known without loading, so it models a STATIC claimer; leaving it at the trait default would have silently re-modelled it as dynamic and made the two native-set tests pass because nothing is refused rather than because refusal works. - `MockEngine::new_dynamic` + two tests binding both halves: `dynamic_claimer_of_native_extension_is_not_probed_or_named` (with a named revert) and `static_claimer_of_native_extension_is_still_named`. E2E (output inspected, `cargo run --bin q2 -- render .`): 3 markdown-only docs load no engine, emit no Q-16-12, and report "1 warning" (Q-16-10 only); the same project plus one computational cell loads the engine and fires Q-16-10/11/12. Codes carry Q-16-10/11/12 with docs pages (error-docs-page-missing requires same-commit). The codes are inert for suppression today: project-scoped diagnostics do not flow through run_pipeline, where DiagnosticPolicy is applied — filed as bd-aow4qio3. Also corrects two stale artifacts: the `// Task: drain registry.diagnostics` marker, and registry.rs's doc comment claiming the vec was already drained. Workspace: 12,637 passed, 0 failed, zero deno skips. Full `cargo xtask verify` green (all 14 steps, incl. WASM leg and clippy -D warnings). No snapshot files touched. Strands: bd-exhbc6h8, bd-7keh8iwn Discovered: bd-aow4qio3 (project-scope suppression), bd-wcmk1fsq (docs sidebar)
Quarto 2 auto-discovers `**/*.qmd` and nothing else. `.md`, `.ipynb`,
percent/spin scripts and engine-contributed extensions all render only when
a `project.render` pattern matches them.
This supersedes the merge runbook's D1, which widened the default pattern
set with one `**/*.<ext>` per statically-claimed extension.
WHY
main's bd-6d2wj4zp already made `.md` opt-in as a deliberate departure from
Quarto 1. D1 then reintroduced the Q1 model for engine extensions only, so
`.md` needed listing but `.echo` did not — hard to explain and harder to
justify.
The Q1 model was confirmed by source audit (2026-08-18): Q1 walks the whole
project and asks each engine to claim what it finds, which for `.py`/`.R`
means OPENING every one of them and regex-matching for `# %%` cells
(core/jupyter/percent.ts:32-45) or a `#' ---` header
(execute/rmd.ts:570-579), at discovery time, on every render. Q1 also
auto-discovers `.md` (execute/markdown.ts:39-42) — so this is not "matching
Q1's .md rule", it is extending an existing q2 divergence to cover every
input type.
Deciding whether a `.py` is a percent script or a `.echo` a document rather
than a fixture requires opening it. That is what lets installing an
extension silently change which files a project renders. One line of
`render:` cannot surprise anyone.
WHAT CHANGED
- the widening loop in `effective_render_patterns` is gone
- `RenderableExtensions::engine_claimed` + accessor deleted; the newtype
collapses back to a flat set. Reverting D1 also reverted the signature
change D1 forced: `effective_render_patterns(user: &[RawGlob])` is main's
original.
- module docs rewritten around the two gates, which is the thing that is
easy to get wrong: gate 1 ("may this extension ever be an input") is
unchanged and still admits engine claims; gate 2 ("is this file in the
render list") is where the default lives. Passing the first does not
imply passing the second.
BEHAVIOR CHANGE BEYOND DISCOVERY
`q2 render a.echo` on an unlisted file now fails with NotInRenderList.
Under D1 auto-discovery put it in `project.files`, so it worked. This
matches `.md` and `_partial.qmd`, which already fail that way — Gordon
confirmed it as consistent. Bound by
`classify_unlisted_echo_file_is_refused`, whose comment records that it
documents the consequence rather than endorsing it: whether q2 should
render an explicitly-named file no pattern selects is an open question (Q1
does, silently, writing output beside the source instead of into the output
dir).
The exclusion message is deliberately NOT improved here. `.ipynb` still
gets the generic hint. Three distinct refusal reasons need three different
answers, and a uniform "add a pattern" message would be actively wrong for
`_partial.qmd` — needs more discussion.
TESTS
Rescued rather than deleted, where a test proved more than one thing:
- `t6` bound that engine extensions obey the same exclusions as `.qmd`
(underscore/dot/output-dir). Still valuable; given an explicit pattern so
it has something to exclude.
- `t8` proved discovery AND conversion end-to-end. Given a `render:` key so
it still proves conversion through the orchestrator.
- the CLI classify test split in two: admitted-when-listed (original
intent, gate 1 works) and refused-when-unlisted (the new behavior).
New: `claimed_extension_is_not_auto_discovered`,
`default_pattern_set_is_qmd_only_regardless_of_claims`, and
`a_positive_user_pattern_replaces_the_default_entirely` — the last renamed
from a D1 boundary case because it now documents a trap: a positive pattern
REPLACES the default, so anyone adding `render:` must keep `**/*.qmd` or
lose their pages.
DOCS
- `render-list.qmd`: a table of what renders by default, and the
replace-not-add trap called out with an example. It also finally says WHY
`.md` is opt-in — the rule and the fix were documented but never the
reason, so a reader arriving from Quarto 1 had no way to tell a
deliberate policy from a missing feature.
- plans 7b and 7c get migration notes. 7c records that auto-discovering
`.ipynb` was considered and REJECTED (Gordon, 2026-08-18) so it is not
re-opened as an artifact of incomplete ipynb support; both note the
failure mode is silent (zero files, no diagnostic) and docs are the
entire mitigation.
- runbook D1 superseded in place, keeping the contradicting Phase C e2e
record as the clearest statement of why the question was live.
Workspace: 12,638 passed, 0 failed, zero deno skips. Full `cargo xtask
verify` green (all 14 steps). No snapshot files.
The error-reference sidebar in docs/_quarto.yml enumerates every error page
by hand. Nothing enforced that the list stayed complete, and it drifted:
153 of 211 pages were listed. Two subsystems had no `- section:` block at
all — `extension` (12 pages) and `crossref` (1) — so every Q-15-* and
Q-16-* page was unreachable by navigation.
The pages render and resolve by direct URL, so no diagnostic shipped a 404
and `cargo xtask lint` stayed green. Only navigation was missing.
Backfill the 58 unlisted entries and add `error-docs-sidebar-unlisted` to
`cargo xtask lint` with three problem classes:
- unlisted page — a page no sidebar entry references
- stale entry — an entry whose page does not exist (dead link)
- out-of-order — entries within a section must ascend by code number,
so appending alphabetically cannot drift the sidebar
into lexicographic order
Section order is deliberately not policed: the 13 historical sections keep
their arbitrary order and `crossref`/`extension` are appended. No canonical
section order has been agreed, and reordering would churn every section for
no reader benefit.
Four of the 58 came from work in flight on this branch — Q-2-50 (the
native-set engine-claim refusal) and Q-16-10/11/12 (engine registry
diagnostics). Both added error pages and stopped at the catalog, which is
precisely the drift this rule exists to catch.
Why the hand-maintained list stays: no automatic alternative exists in
either Quarto. Q2 supports `auto:` in sidebars, but within-section sort
falls back to the document title, not the code. Q1 cannot do it either — it
merges arbitrary front matter into listing items, so `sort: code` is
accepted, but its build-time comparator is lodash orderBy over plain
relational operators, which puts Q-1-10 before Q-1-2. The only mechanism
either offers is a hand-maintained numeric `order:` on every page, which is
a redundant per-page field and was rejected.
Note the sections were already in numeric order; no resequencing was
needed. A sparse section looks lexicographic but is not — `yaml` runs
Q-1-1, Q-1-10, … Q-1-29, Q-1-99 because it has no Q-1-2..Q-1-9. The
docs/_quarto.yml change is therefore purely additive (+62/-0). Because that
makes the ordering check pass trivially on the real tree, it was bound by
injecting a Q-2-9/Q-2-10 swap and confirming it is flagged at the right
line.
Docs: CLAUDE.md gains the rule; docs/errors/README.md gains a step 4 in
"Adding a new page" and documents all three problem classes plus the
sparse-code caveat, so nobody "fixes" yaml into Q-1-1, Q-1-2. Both state
the drift as 153 of 211 — the count on this branch, which carries four more
error pages than main.
Verification: full `cargo xtask verify` green (14 steps, 12182 tests);
`cargo run --bin q2 -- render docs/` showed all 15 sections in the rendered
sidebar.
Found while doing bd-exhbc6h8. Index ordering remains open as bd-otmqu,
which this commit widens by comment: docs/errors/index.qmd currently
renders its Code/Subsystem/Status columns empty on all rows, because
top-level front-matter keys never reach ListingItem at all.
Strand: bd-wcmk1fsq
f8df952..4eaede0, including the v0.23.0 release. Small merge compared to the last one (221cb52: 450 commits, 26 conflicts) — 18 files touched on both sides, 5 conflicted. Trivial: keep both sides ------------------------ `stage/stages/mod.rs` and `tests/integration/main.rs` collided only because the branch's `source_conversion` / `synth_engines_e2e` and main's `tabsets_js` / `tabset_pipeline` sort adjacently. Both kept, in order. pipeline.rs: 24 -> 25 stages --------------------------- main inserted `TabsetsJsStage` into the JS block, so the merged native pipeline carries both new stages and is 25 long: 0 source-conversion (branch) 13 bootstrap-js 1 parse-document 14 clipboard-js 2 metadata-merge 15 tabsets-js (main) 3 language-resolve (main) 16 attribution-generate 4..12 unchanged 17..24 unchanged The unconflicted tail needed hand-renumbering (15..23 -> 16..24): main's indices happened to coincide with the branch's from `attribution-generate` onward, so git saw those lines as identical and left them alone while the insertion shifted them. `build_wasm_html_pipeline` stays DELETED. main kept extending it (language-resolve, a tabsets-js omission assertion); the branch removed it as dead in the SourceType commit. Verified the deletion still holds: zero references remain, and main touched none of the other files that used to call it (`lib.rs`, `document_profile_pipeline.rs`, `language_pipeline.rs`), so their deletions merged without conflict. Q-2-50 collision: the branch's code becomes Q-2-51 -------------------------------------------------- Both sides minted `Q-2-50` and both added `docs/errors/markdown/Q-2-50.qmd` (an add/add conflict). main's is "Doubled curly braces are not supported" (297f5cc), shipped in v0.23.0, so the released code owns the number. The branch's native-set engine-claim refusal is renumbered to `Q-2-51`: - catalog entry re-keyed, `docs_url` updated; - page split: main's text stays at `Q-2-50.qmd`, the branch's moves to `docs/errors/markdown/Q-2-51.qmd`; - 17 references renumbered across `source_conversion.rs`, `discovery.rs`, `engine/traits.rs`, `engine_diagnostics_cli.rs`, and the `Q-16-10` cross-link; - the merge runbook's D5 section gets a supersession note. Its Phase C e2e transcripts still print `Q-2-50` and are deliberately left alone — that is what the tool emitted at the time. Nothing else moves: `Q-2-51` is the next free `Q-2-*` slot. The new sidebar lint earned its keep ------------------------------------ `error-docs-sidebar-unlisted` (added in the previous commit) had real work to do on its first merge: - backfilled main's 4 new pages: `Q-2-50`, `Q-13-9`, `Q-5-29`, `Q-14-5`; - removed a STALE entry: main deleted `docs/errors/theme/Q-14-3.qmd` in the light-dark work (85bc570) while this branch's sidebar still listed it — a dead nav link neither side's tests would have caught, and precisely the failure class the rule was written for. Sidebar entries and pages now agree exactly at 214 each. Counts in CLAUDE.md and docs/errors/README.md updated (the 153-of-211 drift figure is kept as the historical finding, annotated with today's total). ProjectContext test helpers (cfg(test)-only breakage) ----------------------------------------------------- main's new `tabsets_js.rs` and `panel_tabset.rs` build `ProjectContext { .. }` in their unit tests with the 5 fields main knows about. This branch added four more (`registry`, `extensions`, `binary_dependencies`, `tabled_engines`), so both initializers now spread `..Default::default()`, matching the sibling `bootstrap_js.rs` helper they were copied from. Worth noting how this hid: `cargo build --workspace` passed clean. The failures were `cfg(test)`-only and surfaced only under `cargo build --workspace --tests` / clippy — i.e. only at the step that matches CI strictness. No snapshot files changed on either side of this merge.
gordonwoodhull
force-pushed
the
feature/ts-engine-extensions
branch
from
August 18, 2026 16:40
aa177fa to
986929a
Compare
gordonwoodhull
marked this pull request as ready for review
August 18, 2026 16:41
Member
Author
|
Will merge this if tests pass. In theory, no one should see any difference unless they have engine extensions installed. (And we do not bundle the Julia engine yet.) But there were some deep changes to how engines work, so lmk if anything breaks. Will follow up soon with PRs to quarto-marimo and quarto-julia-engine. They should run with warnings with this PR merged, but they need static claiming and handled-language ignoring to be fully compatible. (I also may try a q2 release later if no one does it first.) |
`f3_timeout_poisons_then_transparently_relaunches` failed twice on ubuntu-latest (CI run 32161552230, attempts 1 and 2) while passing on macos-latest in the same run, on the same commit, and having passed on ubuntu before the main merge. ROOT CAUSE `BEHAVE_LAUNCH_MARKER:2` is not produced by Rust. The fixture writes it to stderr via `console.error`; `stderr_loop` then forwards it into `tracing` from a background thread, where `LaunchMarkerCapture` records it. The marker-2 assertion counted it the instant `execute()` returned, with zero tolerance for that forwarding hop — so it raced the forwarder. Its sibling assertion already knew this: the marker-1 check sleeps 200ms first, with a comment explaining exactly why. The marker-2 check simply never got the same treatment. The race is easy to lose on Linux CI because execute-1's abandoned `QUARTO_SLOW` request is STILL RUNNING inside the same live deno process (its 60s branch) when execute-2 issues its `LaunchEngine` — the timeout poisons only the logical instance, deliberately leaving the process and transport alive (`TsEngineHost::write` doc comment; the `Cancelled | Timeout` arm of `TsEngine::execute`'s poison guard). So the process is writing marker-2 to a contended stderr while finishing abandoned work. WHY THE MERGE SURFACED IT Nothing in the merge touched the engine subsystem — the only engine-code change in `aa177fa60..986929a` is a `Q-2-50`→`Q-2-51` doc-comment renumber. The merge added 147 tests, which moved this test from nextest index 9730/12649 to 9795/12796, i.e. into a different (busier) concurrency window on a 4-vCPU runner. Because nextest scheduling is largely deterministic, the same unlucky window recurred on both attempts with near-identical timing (1.428s / 1.434s) — a latent race presenting as a repeatable failure, not a regression. A misreading worth recording: the captured messages show only ONE "engine-host spawned", which looks like proof the relaunch never happened. It isn't. Timeout-poisoning does not respawn the subprocess, so a second spawn is never expected. Only the fixture-side marker is missing. The failing run is also SHORTER than the passing one (1.428s vs 1.491s), because it aborted at the assertion rather than going on to the `r2.expect()` and content checks — the signature of a premature check, not of absent work. THE FIX `LaunchMarkerCapture::wait_for_count_containing(needle, expected, timeout)` — poll at 25ms up to 10s, then settle 200ms and return the final count. The binding is preserved, not weakened: - if the relaunch genuinely never happens, the poll exhausts its window, returns the wrong count, and the assertion still fires. Verified by pointing the poll at a marker that is never emitted: RED after 3s (`1 failed`), rather than passing vacuously. - the trailing settle makes the `== 1` upper bound STRONGER than before: a spurious EXTRA relaunch now has a window to land, which the bare `count_containing()` could equally have missed. The marker-1 check's fixed 200ms sleep is deliberately left alone. It does double duty as the window that gives the adjacent `marker-2 == 0` assertion something to observe; converting it to a fast poll would return in ~25ms and quietly weaken that check. Verification: `behave_engine_e2e` 6/6 pass locally (f3 in 1.887s). Test-only change; no product code touched.
gordonwoodhull
added a commit
that referenced
this pull request
Aug 18, 2026
test-results/.last-run.json came into main with the #416 merge (added by 6fecd8b on the feature branch). It is a Playwright run artifact, so every local test run in this package dirtied the working tree — it blocked a fast-forward pull of main. Add a test-results/ ignore rule matching the two siblings that already have one (hub-client/.gitignore, q2-preview-spa/.gitignore) and drop the file from the index.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration branch for the TypeScript engine-extensions epic. Draft — not ready to merge; open mainly to exercise CI.
Grand plan:
claude-notes/plans/2026-04-16-ts-engine-extensions-subprocess.md. Latest commit reconciles plan status (marks completions, tombstones Plan 8 — diagrams are not engines).