fix(regex): preserve surrogate property scalar semantics

AGENTS.md

@@ -199,6 +199,8 @@
 | 2026-04-30 | (no work lost — recovered) Working tree on `fix/class-trait-tests` was overwritten with master content | Agent ran `git checkout master -- .` to A/B test failures vs master without first snapshotting and without switching branches. Recovery only worked because the changes had already been committed to HEAD: `git restore .` (also a forbidden command on a dirty tree, but safe here because "dirty" was master content, not user work) brought the tree back from HEAD. Correct workflow would have been: stash via `git diff > /tmp/wip.patch`, or use `git worktree add` for the master comparison instead of mutating the current tree. |
 | 2026-04-30 | A full afternoon chasing a phantom "DBIx::Class regression" in `t/76joins.t` / `t/96_is_deteministic_value.t` | Investigative agent launched the test repeatedly under `/usr/bin/time -p ./jperl …` (no `timeout` wrapper). Each hung JVM survived past the agent's lifetime, accumulated as ~14 orphans at 100% CPU each, and starved the active `jcpan` harness — which then SIGKILLed innocent tests after 300 s of no TAP output. Symptom looked exactly like a real perf regression. Fix: always `timeout N ./jperl …` for any potentially-hanging run. |
 | 2026-08-06 | (no source work lost — build recovered) A process cleanup killed the active Gradle test workers, producing exit 137 failures in two shards. | Agent selected Java PIDs from a broad CPU list without first constraining them to stale processes. Recovery: rerun `make` without killing workers; subsequent build completed successfully. Fix: never kill by CPU list alone; identify the exact command and build ownership first. |
+| 2026-08-18 | (no source work lost — corpus rerun required) A generated Unicode corpus read the worktree JAR while a concurrent `make` replaced it, and an interpreter worker failed loading an ICU class. | Agent launched `jperl` corpus workers from the same worktree before its `shadowJar` task had finished. Recovery: let bounded workers finish and rerun both backends after the build stops mutating the artifact. Fix: never run `jperl`/corpus validation from a worktree whose build may replace its JAR; wait for the build or use a verified immutable artifact copy. |
+| 2026-08-18 | (no source work lost — overloaded corpus rerun required) Eight generated TestProp JVMs overlapped another engineer's full build and the five-worker acceptance suite; the 20-minute outer commands ended while exact per-test timeout/JVM descendants remained at high CPU. | Agent treated corpus workers as independent of the two-build limit without accounting for total host load and descendant cleanup. Recovery: identify the exact worktree JAR, test names, PIDs, and parentage; terminate only those orphaned corpus runners/timeouts/JVMs; preserve the unrelated build and acceptance workers. Fix: run this corpus at lower parallelism after builds release their slots, and verify exact descendants after every outer timeout. |
 | 2026-08-17 | (no source work lost — stale workers removed) Failed `make` runs were interrupted after their known Joni failures, but their Gradle unit-shard workers survived and competed with later builds. | Agent sent Ctrl-C to the parent build session before all parallel workers had exited. Recovery: identified stale workers by PID, start time, and shard work directory, terminated only those exact PIDs, and left the current build and sibling repositories untouched. Fix: let failed parallel `make` runs finish naturally, or verify and clean up their exact child PIDs before starting another build. |
 | 2026-08-17 | (no source work lost — CPAN run rerun) A concurrent `make` replaced the development shadow JAR while an active `jcpan` process was spawning a child JVM, causing a transient `ClassNotFoundException`. | Agent waited for another worktree's build but did not wait for the same worktree's bounded CPAN runs before rebuilding `target/perlonjava-5.44.0.jar`. Recovery: let `make` finish and rerun the affected CPAN target. Fix: never rebuild a worktree's development JAR while that worktree has active `jperl` or `jcpan` processes. |
 

dev/design/phase36-regex-parity.md

@@ -41,12 +41,18 @@ baseline, with no per-file pass-count regressions.
 
 ### Migration controls
 
-A temporary developer-only backend selector supports separate Java and Joni
-corpus runs. It must never run both matchers for one operation because callbacks,
-tied variables, `pos()`, and substitutions may have observable side effects.
-Joni is the default matcher; explicit Java mode remains only for differential
-measurement. The selector and Java matching fields are removed at the end of the
-migration.
+A temporary backend selector supports separate Java and Joni corpus runs. It
+must never run both matchers for one operation because callbacks, tied
+variables, `pos()`, and substitutions may have observable side effects. The
+intended migration default is Joni. While the remaining forced-Joni corpus gap
+would otherwise regress the PR 958 acceptance baseline, automatic routing may
+temporarily keep ordinary patterns on Java and require Joni only for executable
+callbacks/dynamic callouts and the advanced constructs recognized by
+`requiresJoniBackend`. Forced-Joni differential coverage remains mandatory.
+This progressive-routing workaround is not a final architecture, does not
+satisfy the Phase 1 or Phase 5 exit criteria, and must be removed at the
+earliest evidence-backed point. The selector and all Java matching fields are
+removed at the end of the migration.
 
 ### Preprocessing boundary
 
@@ -288,6 +294,19 @@ The property corpus therefore reaches 153,882/167,506 and the complete
 property-plus-boundary evidence reaches 393,748/407,372, leaving 13,624
 property assertions.
 
+Draft PR #1049 implements the shared marker-aware surrogate property
+renderer. Its standard-Perl-first focused oracle passes 27/27 on standard
+Perl, JVM, and interpreter; direct Joni tests, packaging verification, and a
+warning-free 4m58s `make` pass. Positive and complemented properties,
+ordinary and negated classes, captures, substitution, and `/g` now consume
+U+D800..U+DFFF markers as one Perl scalar. The authoritative stable-JAR,
+two-worker chunks 01–04 map passes 154,132/167,506 on both JVM and interpreter
+with exact numbered-test identity, 250 gains, zero losses, and no missing
+assertions. Combined with the complete boundary corpus, generated evidence is
+393,998/407,372 and the residual property surface is 13,374 assertions. The
+first two resource-collided attempts remain explicitly discarded and do not
+contribute to these totals.
+
 Lexical `use bytes` now compiles non-ASCII substitution patterns with a
 single-byte Joni encoding while preserving upgraded, byte-backed, and compiled
 `qr//` source provenance. The focused oracle passes 12/12 on system Perl, JVM,
@@ -443,6 +462,16 @@ focused precedence, class-negation, and bare-scx reducers pass 7/7, 8/8, and
 regressions and exact JVM/interpreter counts, raising current generated
 evidence to 377,602/407,367.
 
+The independently validated bare-Script alias follow-up is published as
+`296f240c2`. It applies Perl's ASCII-loose bare `Is` Script shortcut without
+changing explicit assignment, generated-data, or precedence policy. Its
+standard-Perl, JVM, and interpreter oracle passes 12/12; protected boundaries
+remain exact; warning-free `make`, Joni, and packaging pass. On its exact old
+base, the pinned target map gains 3,832 assertions in chunk 03 and 1,232 in
+chunk 04, exactly +5,064 with zero missing/failing target IDs and exact backend
+identity. Aggregate totals remain deferred until this commit is rebased onto
+the current residual stack.
+
 `Grapheme_Cluster_Break`/`gcb`, `Sentence_Break`/`sb`,
 `Word_Break`/`wb`, and `Line_Break`/`lb` assignments now resolve every pinned
 Unicode 17 value. Short, long, loose, colon, wildcard, missing-value, and exact
@@ -546,6 +575,16 @@ Strict-regex source policy and Perl code points above U+10FFFF remain explicit
 frontend/representation debt, so the forced-Java underscore compatibility pass
 is retained for now.
 
+The final pre-workaround PR #1042 integration run completed at
+505,864/561,385 against PR 958's 269,171/324,252. Its exact file comparison
+reports 41 regression files and a 5,035-assertion negative sum. The leading
+pure-Joni blockers are 4,298 assertions behind the first aborts in
+`pat_advanced{,_thr}.t` and `pat{,_thr}.t`, followed by 224 in `reg_mesg.t`
+and 259 across seven `regexp*` variants; `regex_sets.t` is now only one
+assertion below baseline. These counts prioritize the cleanup after temporary
+progressive routing: unblock the four `pat*` files first, then diagnostics and
+the ordinary `regexp*` matrix. They are not accepted limitations.
+
 ### Completed Phases
 
 - [x] Phase 0: Reproducible differential baseline (2026-08-17)
@@ -559,6 +598,17 @@ is retained for now.
 - [ ] Phase 1: Joni ordinary-pattern parity (implementation substantially
   complete; forced Java/Joni corpus comparison remains)
   - [x] Added the temporary backend selector and made Joni the default.
+  - [x] Added the explicitly temporary progressive acceptance route in PR
+    #1042 (`7173d9bff`): default/`auto` keeps ordinary patterns on Java while
+    every existing `requiresJoniBackend` construct still forces Joni. Explicit
+    Joni mode and the unchanged unit corpus remain forced-Joni; its focused
+    nine-file gate gained six passes with no regressions and `make` passed
+    warning-free in 4m04s.
+  - [ ] Remove the temporary progressive-routing acceptance workaround after
+    closing its forced-Joni regressions. Until then automatic routing keeps
+    ordinary patterns on Java while executable closures, dynamic callouts,
+    recursion/conditions, lookbehind, and control verbs require Joni. This is
+    explicitly cleanup debt and does not weaken the all-Joni exit criterion.
   - [x] Routed ordinary matching, substitution, and split through the selected
     backend without per-operation fallback.
   - [x] Completed the forced-Java/JVM 80-file leg and identified the
@@ -757,6 +807,11 @@ is retained for now.
     aliases, precedence, and ordinary character-class complements. The focused oracle
     passes 95/95; chunks 01–04 gain 8,140 assertions with zero numbered
     regressions and exact backend counts.
+  - [x] Validated bare ASCII-loose `Is` Script aliases independently
+    (`296f240c2`): 12/12 on standard Perl, JVM, and interpreter, protected
+    boundaries exact, warning-free `make`, and an exact pinned target-map gain
+    of 5,064 with zero losses or backend differences. Rebase/integration onto
+    the current residual stack remains pending before aggregate totals change.
   - [x] Generated and integrated pinned Unicode 17.0 GCB/SB/WB/LB property
     values with official aliases, loose and wildcard policy, exact `Is`
     rejection rules, and complete defaults. The focused oracle passes 92/92;
@@ -815,9 +870,21 @@ is retained for now.
     with only nine isolated-surrogate TODOs; chunk 01 gains exactly 2,765
     numbered assertions with zero losses and exact 41,848-assertion backend
     identity, reaching 153,882/167,506 properties.
-  - [ ] Render resolved surrogate property subsets as Perl scalar markers so
+  - [x] Render resolved surrogate property subsets as Perl scalar markers so
     positive, complemented, class, capture, substitution, and `/g` membership
-    preserve the exact U+D800..U+DFFF truth table.
+    preserve the exact U+D800..U+DFFF truth table. The focused oracle passes
+    27/27 on standard Perl, JVM, and interpreter; warning-free `make`, direct
+    Joni, and packaging gates pass in draft PR #1049.
+  - [x] Completed the authoritative stable-JAR chunks 01–04 map for the
+    surrogate renderer at 154,132/167,506 on both backends: exactly 250 gains,
+    zero losses, no missing numbered assertions, and exact backend identity.
+    Generated property-plus-boundary evidence is 393,998/407,372.
+  - [ ] Prevent unanchored native Joni properties/classes that contain no
+    translated surrogate range from beginning inside the visible payload of
+    an internal scalar marker. The isolated reducer at
+    `/tmp/phase36-native-property-marker-boundary.t` passes 2/2 on standard
+    Perl and 1/2 on PerlOnJava; this is separate from PR #1049's
+    surrogate-bearing translated-class renderer.
   - [x] Integrated native Perl `\v`/`\V` dispatch inside and outside character
     classes (`1eff1db97`, integrated as `6328935cd`). The focused oracle passes
     92/92 and unchanged `reg_posixcc.t` passes 2,560/2,560 on both backends.
@@ -874,21 +941,27 @@ is retained for now.
 3. Preserve draft PR #1046's completed combined QC/HST, five-family enumerated,
    and InPC/InSC map plus the follow-on mechanical-cleanup, Identifier,
    `kEH_Core`, and Block/Blk wildcard checkpoints. Integrate the independently
-   validated importer-owned Unikemet snapshot, then complete the shared
-   marker-aware isolated-surrogate property renderer. Preserve pinned Perl 5.44
-   acceptance and rejection semantics rather than inheriting host ICU breadth.
-   The post-Block generated residual is 13,624 assertions: 12,797
-   alias/precedence, 476 wildcard, 346 diagnostic/value-policy, and five shared
-   runtime assertions.
+   validated importer-owned Unikemet snapshot, preserve PR #1049's completed
+   stable-JAR generated map, and integrate the separate native-property
+   marker-boundary reducer. Preserve pinned Perl 5.44 acceptance and rejection semantics
+   rather than inheriting host ICU breadth.
+   The post-surrogate-renderer generated residual is 13,374 assertions; retain
+   the exact numbered differential as the classification source for subsequent
+   alias/precedence, wildcard, diagnostic/value-policy, and shared-runtime work.
    Keep native `\v`/`\V` exact at 2,560/2,560 in `reg_posixcc.t`.
 4. Rerun generated property chunks 01–04 on both backends with the classified
    600-second bound and retain complete TAP/JSON. After the two fatal roots and
    native line boundaries integrate, refresh the complete forced-Joni 80-file
    corpus and apply the no-regression gate against Phase 0 and PR 958.
-5. Audit every `RegexPreprocessor` rule against the final ownership boundary.
-   Move matcher semantics into Joni, retain only source-policy scanning, delete
-   Java-only rewrites and compiled-pattern variants, and remove the temporary
-   Java backend selector after the performance gate passes.
+5. Continue the forced-Joni remediation immediately after the temporary
+   progressive-routing integration gate. Prioritize the ordinary-pattern
+   regressions hidden by automatic Java routing, remove that workaround as soon
+   as the PR 958 no-regression gate permits, and do not mark Phase 1 complete
+   while it remains. Audit every `RegexPreprocessor` rule against the final
+   ownership boundary; move matcher semantics into Joni, retain only
+   source-policy scanning, delete Java-only rewrites and compiled-pattern
+   variants, and remove the temporary backend selector after the performance
+   gate passes.
 6. Reconcile `docs/reference/feature-matrix.md` with the final corpus; update
    `dev/implementation/regex.md` and `docs/design/joni-callout-fork.md` to the
    as-implemented architecture and review both for clarity and structure.

docs/reference/feature-matrix.md

@@ -398,8 +398,8 @@ my @copy = @{$z};         # ERROR
 - ✅  **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables.
 - ✅  **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni.
 - ✅  **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni.
-- 🟡  **Extended Unicode Regex Features**: Complete pinned Script, Script_Extensions, Block, break-property, 51 core binary-property, seven specialized binary-property, QC/HST, five residual enumerated-property, InPC/InSC, Identifier_Status, Identifier_Type, and provisional `kEH_Core` sets, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, Numeric_Value, Joining_Group, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Block/Blk value wildcards preserve official aliases, Perl-loose matching, colon/slash delimiters, warning order, and the `InKana` collision policy. Current generated evidence is 393,748/407,372 on both backends: chunks 01–04 pass 153,882/167,506 and the complete boundary corpus passes 239,866/239,866.
-- 🟡  **Remaining generated Unicode property surface**: The measured post-Block-wildcard map has 13,624 residual assertions: 12,797 alias/precedence, 476 wildcard, 346 diagnostic/value-policy, and five shared boundary/runtime assertions. The shared marker-aware isolated-surrogate renderer is the next cross-cutting property slice. The focused Block oracle passes 38/47 per backend and the focused `kEH_Core` oracle passes 167/169 because isolated-surrogate property membership remains pending.
+- 🟡  **Extended Unicode Regex Features**: Complete pinned Script, Script_Extensions, Block, break-property, 51 core binary-property, seven specialized binary-property, QC/HST, five residual enumerated-property, InPC/InSC, Identifier_Status, Identifier_Type, and provisional `kEH_Core` sets, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, Numeric_Value, Joining_Group, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Block/Blk value wildcards preserve official aliases, Perl-loose matching, colon/slash delimiters, warning order, and the `InKana` collision policy. The marker-aware isolated-surrogate renderer raises current generated evidence to 393,998/407,372 on both backends: chunks 01–04 pass 154,132/167,506 with exact numbered-test identity and the complete boundary corpus passes 239,866/239,866.
+- 🟡  **Remaining generated Unicode property surface**: The authoritative post-surrogate-renderer map has 13,374 residual assertions. The renderer gains 250 assertions over the preceding family-specific baselines with zero losses and no missing numbered assertions; remaining failures are alias/precedence, wildcard, diagnostic/value-policy, and five shared boundary/runtime assertions tracked by the generated differential.
 - 🟡  **Extended regex sets**: `(?[...])` set expressions are implemented through the frontend extended-character-class translator, but full Perl parser and diagnostic parity is not complete in native Joni.
 - ✅  **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter.
 - ✅  **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter.