diff --git a/AGENTS.md b/AGENTS.md index 4c3e61c9f5..723b600468 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -199,6 +199,8 @@ | 2026-04-30 | (no work lost — recovered) Working tree on `fix/class-trait-tests` was overwritten with master content | Agent ran `git checkout master -- .` to A/B test failures vs master without first snapshotting and without switching branches. Recovery only worked because the changes had already been committed to HEAD: `git restore .` (also a forbidden command on a dirty tree, but safe here because "dirty" was master content, not user work) brought the tree back from HEAD. Correct workflow would have been: stash via `git diff > /tmp/wip.patch`, or use `git worktree add` for the master comparison instead of mutating the current tree. | | 2026-04-30 | A full afternoon chasing a phantom "DBIx::Class regression" in `t/76joins.t` / `t/96_is_deteministic_value.t` | Investigative agent launched the test repeatedly under `/usr/bin/time -p ./jperl …` (no `timeout` wrapper). Each hung JVM survived past the agent's lifetime, accumulated as ~14 orphans at 100% CPU each, and starved the active `jcpan` harness — which then SIGKILLed innocent tests after 300 s of no TAP output. Symptom looked exactly like a real perf regression. Fix: always `timeout N ./jperl …` for any potentially-hanging run. | | 2026-08-06 | (no source work lost — build recovered) A process cleanup killed the active Gradle test workers, producing exit 137 failures in two shards. | Agent selected Java PIDs from a broad CPU list without first constraining them to stale processes. Recovery: rerun `make` without killing workers; subsequent build completed successfully. Fix: never kill by CPU list alone; identify the exact command and build ownership first. | +| 2026-08-18 | (no source work lost — corpus rerun required) A generated Unicode corpus read the worktree JAR while a concurrent `make` replaced it, and an interpreter worker failed loading an ICU class. | Agent launched `jperl` corpus workers from the same worktree before its `shadowJar` task had finished. Recovery: let bounded workers finish and rerun both backends after the build stops mutating the artifact. Fix: never run `jperl`/corpus validation from a worktree whose build may replace its JAR; wait for the build or use a verified immutable artifact copy. | +| 2026-08-18 | (no source work lost — overloaded corpus rerun required) Eight generated TestProp JVMs overlapped another engineer's full build and the five-worker acceptance suite; the 20-minute outer commands ended while exact per-test timeout/JVM descendants remained at high CPU. | Agent treated corpus workers as independent of the two-build limit without accounting for total host load and descendant cleanup. Recovery: identify the exact worktree JAR, test names, PIDs, and parentage; terminate only those orphaned corpus runners/timeouts/JVMs; preserve the unrelated build and acceptance workers. Fix: run this corpus at lower parallelism after builds release their slots, and verify exact descendants after every outer timeout. | | 2026-08-17 | (no source work lost — stale workers removed) Failed `make` runs were interrupted after their known Joni failures, but their Gradle unit-shard workers survived and competed with later builds. | Agent sent Ctrl-C to the parent build session before all parallel workers had exited. Recovery: identified stale workers by PID, start time, and shard work directory, terminated only those exact PIDs, and left the current build and sibling repositories untouched. Fix: let failed parallel `make` runs finish naturally, or verify and clean up their exact child PIDs before starting another build. | | 2026-08-17 | (no source work lost — CPAN run rerun) A concurrent `make` replaced the development shadow JAR while an active `jcpan` process was spawning a child JVM, causing a transient `ClassNotFoundException`. | Agent waited for another worktree's build but did not wait for the same worktree's bounded CPAN runs before rebuilding `target/perlonjava-5.44.0.jar`. Recovery: let `make` finish and rerun the affected CPAN target. Fix: never rebuild a worktree's development JAR while that worktree has active `jperl` or `jcpan` processes. | diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index a45130db75..1654507cab 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -41,12 +41,18 @@ baseline, with no per-file pass-count regressions. ### Migration controls -A temporary developer-only backend selector supports separate Java and Joni -corpus runs. It must never run both matchers for one operation because callbacks, -tied variables, `pos()`, and substitutions may have observable side effects. -Joni is the default matcher; explicit Java mode remains only for differential -measurement. The selector and Java matching fields are removed at the end of the -migration. +A temporary backend selector supports separate Java and Joni corpus runs. It +must never run both matchers for one operation because callbacks, tied +variables, `pos()`, and substitutions may have observable side effects. The +intended migration default is Joni. While the remaining forced-Joni corpus gap +would otherwise regress the PR 958 acceptance baseline, automatic routing may +temporarily keep ordinary patterns on Java and require Joni only for executable +callbacks/dynamic callouts and the advanced constructs recognized by +`requiresJoniBackend`. Forced-Joni differential coverage remains mandatory. +This progressive-routing workaround is not a final architecture, does not +satisfy the Phase 1 or Phase 5 exit criteria, and must be removed at the +earliest evidence-backed point. The selector and all Java matching fields are +removed at the end of the migration. ### Preprocessing boundary @@ -288,6 +294,19 @@ The property corpus therefore reaches 153,882/167,506 and the complete property-plus-boundary evidence reaches 393,748/407,372, leaving 13,624 property assertions. +Draft PR #1049 implements the shared marker-aware surrogate property +renderer. Its standard-Perl-first focused oracle passes 27/27 on standard +Perl, JVM, and interpreter; direct Joni tests, packaging verification, and a +warning-free 4m58s `make` pass. Positive and complemented properties, +ordinary and negated classes, captures, substitution, and `/g` now consume +U+D800..U+DFFF markers as one Perl scalar. The authoritative stable-JAR, +two-worker chunks 01–04 map passes 154,132/167,506 on both JVM and interpreter +with exact numbered-test identity, 250 gains, zero losses, and no missing +assertions. Combined with the complete boundary corpus, generated evidence is +393,998/407,372 and the residual property surface is 13,374 assertions. The +first two resource-collided attempts remain explicitly discarded and do not +contribute to these totals. + Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled `qr//` source provenance. The focused oracle passes 12/12 on system Perl, JVM, @@ -443,6 +462,16 @@ focused precedence, class-negation, and bare-scx reducers pass 7/7, 8/8, and regressions and exact JVM/interpreter counts, raising current generated evidence to 377,602/407,367. +The independently validated bare-Script alias follow-up is published as +`296f240c2`. It applies Perl's ASCII-loose bare `Is` Script shortcut without +changing explicit assignment, generated-data, or precedence policy. Its +standard-Perl, JVM, and interpreter oracle passes 12/12; protected boundaries +remain exact; warning-free `make`, Joni, and packaging pass. On its exact old +base, the pinned target map gains 3,832 assertions in chunk 03 and 1,232 in +chunk 04, exactly +5,064 with zero missing/failing target IDs and exact backend +identity. Aggregate totals remain deferred until this commit is rebased onto +the current residual stack. + `Grapheme_Cluster_Break`/`gcb`, `Sentence_Break`/`sb`, `Word_Break`/`wb`, and `Line_Break`/`lb` assignments now resolve every pinned Unicode 17 value. Short, long, loose, colon, wildcard, missing-value, and exact @@ -546,6 +575,16 @@ Strict-regex source policy and Perl code points above U+10FFFF remain explicit frontend/representation debt, so the forced-Java underscore compatibility pass is retained for now. +The final pre-workaround PR #1042 integration run completed at +505,864/561,385 against PR 958's 269,171/324,252. Its exact file comparison +reports 41 regression files and a 5,035-assertion negative sum. The leading +pure-Joni blockers are 4,298 assertions behind the first aborts in +`pat_advanced{,_thr}.t` and `pat{,_thr}.t`, followed by 224 in `reg_mesg.t` +and 259 across seven `regexp*` variants; `regex_sets.t` is now only one +assertion below baseline. These counts prioritize the cleanup after temporary +progressive routing: unblock the four `pat*` files first, then diagnostics and +the ordinary `regexp*` matrix. They are not accepted limitations. + ### Completed Phases - [x] Phase 0: Reproducible differential baseline (2026-08-17) @@ -559,6 +598,17 @@ is retained for now. - [ ] Phase 1: Joni ordinary-pattern parity (implementation substantially complete; forced Java/Joni corpus comparison remains) - [x] Added the temporary backend selector and made Joni the default. + - [x] Added the explicitly temporary progressive acceptance route in PR + #1042 (`7173d9bff`): default/`auto` keeps ordinary patterns on Java while + every existing `requiresJoniBackend` construct still forces Joni. Explicit + Joni mode and the unchanged unit corpus remain forced-Joni; its focused + nine-file gate gained six passes with no regressions and `make` passed + warning-free in 4m04s. + - [ ] Remove the temporary progressive-routing acceptance workaround after + closing its forced-Joni regressions. Until then automatic routing keeps + ordinary patterns on Java while executable closures, dynamic callouts, + recursion/conditions, lookbehind, and control verbs require Joni. This is + explicitly cleanup debt and does not weaken the all-Joni exit criterion. - [x] Routed ordinary matching, substitution, and split through the selected backend without per-operation fallback. - [x] Completed the forced-Java/JVM 80-file leg and identified the @@ -757,6 +807,11 @@ is retained for now. aliases, precedence, and ordinary character-class complements. The focused oracle passes 95/95; chunks 01–04 gain 8,140 assertions with zero numbered regressions and exact backend counts. + - [x] Validated bare ASCII-loose `Is` Script aliases independently + (`296f240c2`): 12/12 on standard Perl, JVM, and interpreter, protected + boundaries exact, warning-free `make`, and an exact pinned target-map gain + of 5,064 with zero losses or backend differences. Rebase/integration onto + the current residual stack remains pending before aggregate totals change. - [x] Generated and integrated pinned Unicode 17.0 GCB/SB/WB/LB property values with official aliases, loose and wildcard policy, exact `Is` rejection rules, and complete defaults. The focused oracle passes 92/92; @@ -815,9 +870,21 @@ is retained for now. with only nine isolated-surrogate TODOs; chunk 01 gains exactly 2,765 numbered assertions with zero losses and exact 41,848-assertion backend identity, reaching 153,882/167,506 properties. - - [ ] Render resolved surrogate property subsets as Perl scalar markers so + - [x] Render resolved surrogate property subsets as Perl scalar markers so positive, complemented, class, capture, substitution, and `/g` membership - preserve the exact U+D800..U+DFFF truth table. + preserve the exact U+D800..U+DFFF truth table. The focused oracle passes + 27/27 on standard Perl, JVM, and interpreter; warning-free `make`, direct + Joni, and packaging gates pass in draft PR #1049. + - [x] Completed the authoritative stable-JAR chunks 01–04 map for the + surrogate renderer at 154,132/167,506 on both backends: exactly 250 gains, + zero losses, no missing numbered assertions, and exact backend identity. + Generated property-plus-boundary evidence is 393,998/407,372. + - [ ] Prevent unanchored native Joni properties/classes that contain no + translated surrogate range from beginning inside the visible payload of + an internal scalar marker. The isolated reducer at + `/tmp/phase36-native-property-marker-boundary.t` passes 2/2 on standard + Perl and 1/2 on PerlOnJava; this is separate from PR #1049's + surrogate-bearing translated-class renderer. - [x] Integrated native Perl `\v`/`\V` dispatch inside and outside character classes (`1eff1db97`, integrated as `6328935cd`). The focused oracle passes 92/92 and unchanged `reg_posixcc.t` passes 2,560/2,560 on both backends. @@ -874,21 +941,27 @@ is retained for now. 3. Preserve draft PR #1046's completed combined QC/HST, five-family enumerated, and InPC/InSC map plus the follow-on mechanical-cleanup, Identifier, `kEH_Core`, and Block/Blk wildcard checkpoints. Integrate the independently - validated importer-owned Unikemet snapshot, then complete the shared - marker-aware isolated-surrogate property renderer. Preserve pinned Perl 5.44 - acceptance and rejection semantics rather than inheriting host ICU breadth. - The post-Block generated residual is 13,624 assertions: 12,797 - alias/precedence, 476 wildcard, 346 diagnostic/value-policy, and five shared - runtime assertions. + validated importer-owned Unikemet snapshot, preserve PR #1049's completed + stable-JAR generated map, and integrate the separate native-property + marker-boundary reducer. Preserve pinned Perl 5.44 acceptance and rejection semantics + rather than inheriting host ICU breadth. + The post-surrogate-renderer generated residual is 13,374 assertions; retain + the exact numbered differential as the classification source for subsequent + alias/precedence, wildcard, diagnostic/value-policy, and shared-runtime work. Keep native `\v`/`\V` exact at 2,560/2,560 in `reg_posixcc.t`. 4. Rerun generated property chunks 01–04 on both backends with the classified 600-second bound and retain complete TAP/JSON. After the two fatal roots and native line boundaries integrate, refresh the complete forced-Joni 80-file corpus and apply the no-regression gate against Phase 0 and PR 958. -5. Audit every `RegexPreprocessor` rule against the final ownership boundary. - Move matcher semantics into Joni, retain only source-policy scanning, delete - Java-only rewrites and compiled-pattern variants, and remove the temporary - Java backend selector after the performance gate passes. +5. Continue the forced-Joni remediation immediately after the temporary + progressive-routing integration gate. Prioritize the ordinary-pattern + regressions hidden by automatic Java routing, remove that workaround as soon + as the PR 958 no-regression gate permits, and do not mark Phase 1 complete + while it remains. Audit every `RegexPreprocessor` rule against the final + ownership boundary; move matcher semantics into Joni, retain only + source-policy scanning, delete Java-only rewrites and compiled-pattern + variants, and remove the temporary backend selector after the performance + gate passes. 6. Reconcile `docs/reference/feature-matrix.md` with the final corpus; update `dev/implementation/regex.md` and `docs/design/joni-callout-fork.md` to the as-implemented architecture and review both for clarity and structure. diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index 135103832a..091c410758 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -398,8 +398,8 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Complete pinned Script, Script_Extensions, Block, break-property, 51 core binary-property, seven specialized binary-property, QC/HST, five residual enumerated-property, InPC/InSC, Identifier_Status, Identifier_Type, and provisional `kEH_Core` sets, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, Numeric_Value, Joining_Group, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Block/Blk value wildcards preserve official aliases, Perl-loose matching, colon/slash delimiters, warning order, and the `InKana` collision policy. Current generated evidence is 393,748/407,372 on both backends: chunks 01–04 pass 153,882/167,506 and the complete boundary corpus passes 239,866/239,866. -- 🟡 **Remaining generated Unicode property surface**: The measured post-Block-wildcard map has 13,624 residual assertions: 12,797 alias/precedence, 476 wildcard, 346 diagnostic/value-policy, and five shared boundary/runtime assertions. The shared marker-aware isolated-surrogate renderer is the next cross-cutting property slice. The focused Block oracle passes 38/47 per backend and the focused `kEH_Core` oracle passes 167/169 because isolated-surrogate property membership remains pending. +- 🟡 **Extended Unicode Regex Features**: Complete pinned Script, Script_Extensions, Block, break-property, 51 core binary-property, seven specialized binary-property, QC/HST, five residual enumerated-property, InPC/InSC, Identifier_Status, Identifier_Type, and provisional `kEH_Core` sets, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, Numeric_Value, Joining_Group, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Block/Blk value wildcards preserve official aliases, Perl-loose matching, colon/slash delimiters, warning order, and the `InKana` collision policy. The marker-aware isolated-surrogate renderer raises current generated evidence to 393,998/407,372 on both backends: chunks 01–04 pass 154,132/167,506 with exact numbered-test identity and the complete boundary corpus passes 239,866/239,866. +- 🟡 **Remaining generated Unicode property surface**: The authoritative post-surrogate-renderer map has 13,374 residual assertions. The renderer gains 250 assertions over the preceding family-specific baselines with zero losses and no missing numbered assertions; remaining failures are alias/precedence, wildcard, diagnostic/value-policy, and five shared boundary/runtime assertions tracked by the generated differential. - 🟡 **Extended regex sets**: `(?[...])` set expressions are implemented through the frontend extended-character-class translator, but full Perl parser and diagnostic parity is not complete in native Joni. - ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. - ✅ **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter. diff --git a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java index e9a39fad46..48affee202 100644 --- a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java +++ b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java @@ -19,6 +19,7 @@ import java.nio.charset.StandardCharsets; import java.util.Iterator; +import java.util.ArrayList; import java.util.LinkedHashMap; import java.util.Map; import java.util.List; @@ -34,6 +35,8 @@ final class JoniRegexPattern { private static final Map INPUT_ENCODINGS = new WeakHashMap<>(); private static final Map BYTE_INPUT_ENCODINGS = new WeakHashMap<>(); + private static final String INTERNAL_SCALAR_BOUNDARY_GUARD = + internalScalarBoundaryGuard(); // Ruby syntax defaults \w to ASCII even for a Unicode encoding. Perl's // default and /u modes use Unicode character classes; /a adds ASCII_RANGE @@ -211,8 +214,13 @@ private static UserPropertyTranslation translateUserDefinedProperties( } if ((frontendProperty || scriptExtensions || perlBuiltInAlias) && standardClassBracketDepth > 0) { - translated.append(UnicodeResolver.translateUnicodePropertyForCharClass( - property, pattern.charAt(i + 1) == 'P')); + String propertyClass = UnicodeResolver.translateUnicodePropertyForCharClass( + property, pattern.charAt(i + 1) == 'P'); + if (propertyClass.startsWith("[") && propertyClass.endsWith("]")) { + translated.append(propertyClass, 1, propertyClass.length() - 1); + } else { + translated.append(propertyClass); + } i = end; continue; } @@ -319,6 +327,7 @@ static String translatePattern(String pattern) { private static String translatePattern(String pattern, RegexFlags flags, int trustedCalloutCount) { pattern = translateDefineBlocks(pattern); + pattern = translateInternalScalarClassMembers(pattern); StringBuilder out = new StringBuilder(pattern.length() + 16); boolean escaped = false; boolean inClass = false; @@ -495,6 +504,255 @@ private static String translatePattern(String pattern, RegexFlags flags, return out.toString(); } + /** + * Perl scalar strings can contain surrogate and beyond-Unicode values. + * They are represented internally as {@code U+FFFD}, which Joni can + * match as ordinary text, but not as one logical character. Lift those + * members and surrogate ranges into alternatives that consume a complete + * internal marker. For complemented classes, exclude selected marker + * payloads while still consuming every other marker atomically. + */ + private static String translateInternalScalarClassMembers(String pattern) { + StringBuilder translated = new StringBuilder(pattern.length()); + boolean escaped = false; + for (int i = 0; i < pattern.length(); i++) { + char ch = pattern.charAt(i); + if (escaped) { + translated.append(ch); + escaped = false; + continue; + } + if (ch == '\\') { + translated.append(ch); + escaped = true; + continue; + } + if (ch != '[') { + translated.append(ch); + continue; + } + + int close = findStandardClassClose(pattern, i + 1); + if (close < 0) { + translated.append(ch); + continue; + } + String replacement = translateInternalScalarClassContent( + pattern.substring(i + 1, close)); + if (replacement == null) { + translated.append(pattern, i, close + 1); + } else { + translated.append(replacement); + } + i = close; + } + return translated.toString(); + } + + private static int findStandardClassClose(String pattern, int start) { + boolean escaped = false; + boolean leading = true; + int posixDepth = 0; + for (int i = start; i < pattern.length(); i++) { + char ch = pattern.charAt(i); + if (escaped) { + escaped = false; + leading = false; + continue; + } + if (ch == '\\') { + escaped = true; + leading = false; + continue; + } + if (ch == '[' && i + 1 < pattern.length() + && (pattern.charAt(i + 1) == ':' + || pattern.charAt(i + 1) == '.' + || pattern.charAt(i + 1) == '=')) { + posixDepth++; + leading = false; + continue; + } + if (ch == ']' && posixDepth > 0) { + posixDepth--; + continue; + } + if (ch == ']' && leading) { + leading = false; + continue; + } + if (ch == ']') return i; + if (ch != '^' || !leading) leading = false; + } + return -1; + } + + private record HexEscape(long value, int endExclusive) {} + + private static String translateInternalScalarClassContent(String content) { + boolean negated = content.startsWith("^"); + int contentStart = negated ? 1 : 0; + StringBuilder retained = new StringBuilder(content.length()); + List exactMarkers = new ArrayList<>(); + List surrogateRanges = new ArrayList<>(); + for (int i = contentStart; i < content.length();) { + if (content.startsWith("\\\\", i)) { + retained.append("\\\\"); + i += 2; + continue; + } + HexEscape first = parseHexEscape(content, i); + if (first != null) { + HexEscape last = first.endExclusive() < content.length() + && content.charAt(first.endExclusive()) == '-' + ? parseHexEscape(content, first.endExclusive() + 1) : null; + if (last != null && first.value() <= last.value() + && first.value() <= 0x10FFFFL && last.value() <= 0x10FFFFL + && first.value() <= Character.MAX_SURROGATE + && last.value() >= Character.MIN_SURROGATE) { + int surrogateStart = (int) Math.max( + first.value(), Character.MIN_SURROGATE); + int surrogateEnd = (int) Math.min( + last.value(), Character.MAX_SURROGATE); + surrogateRanges.add(new int[] {surrogateStart, surrogateEnd}); + if (first.value() < Character.MIN_SURROGATE) { + appendHexClassRange(retained, first.value(), + Character.MIN_SURROGATE - 1L); + } + if (last.value() > Character.MAX_SURROGATE) { + appendHexClassRange(retained, + Character.MAX_SURROGATE + 1L, last.value()); + } + i = last.endExclusive(); + continue; + } + if (isSurrogate(first.value())) { + int value = (int) first.value(); + surrogateRanges.add(new int[] {value, value}); + i = first.endExclusive(); + continue; + } + if (Long.compareUnsigned(first.value(), 0x10FFFFL) > 0 + && last == null) { + exactMarkers.add(Long.toUnsignedString(first.value(), 16) + .toUpperCase(java.util.Locale.ROOT)); + i = first.endExclusive(); + continue; + } + } + retained.append(content.charAt(i++)); + } + if (exactMarkers.isEmpty() && surrogateRanges.isEmpty()) return null; + + List payloads = new ArrayList<>( + exactMarkers.size() + surrogateRanges.size()); + payloads.addAll(exactMarkers); + for (int[] range : surrogateRanges) { + payloads.add(fixedWidthHexRange(range[0], range[1], 4)); + } + String payload = payloads.size() == 1 + ? payloads.get(0) : "(?:" + String.join("|", payloads) + ")"; + String anyMarker = "\\x{FFFD}<[0-9A-F]+>"; + String markerBoundary = INTERNAL_SCALAR_BOUNDARY_GUARD; + + List alternatives = new ArrayList<>(2); + if (negated) { + alternatives.add("\\x{FFFD}<(?!(?:" + payload + ")>)[0-9A-F]+>"); + StringBuilder ordinary = new StringBuilder("(?!") + .append(anyMarker).append(')'); + if (!retained.isEmpty()) { + ordinary.append("(?![").append(retained).append("])" ); + } + alternatives.add(markerBoundary + ordinary.append("[\\s\\S]").toString()); + } else { + alternatives.add("\\x{FFFD}<" + payload + ">"); + if (!retained.isEmpty()) { + alternatives.add(markerBoundary + "(?!" + anyMarker + ")[" + + retained + "]"); + } + } + return alternatives.size() == 1 ? alternatives.get(0) + : "(?:" + String.join("|", alternatives) + ")"; + } + + private static HexEscape parseHexEscape(String content, int offset) { + if (offset < 0 || !content.startsWith("\\x{", offset)) return null; + int close = content.indexOf('}', offset + 3); + if (close <= offset + 3) return null; + try { + return new HexEscape(Long.parseUnsignedLong( + content.substring(offset + 3, close), 16), close + 1); + } catch (NumberFormatException ignored) { + // Let Joni produce the normal malformed-escape diagnostic. + return null; + } + } + + private static boolean isSurrogate(long value) { + return value >= Character.MIN_SURROGATE + && value <= Character.MAX_SURROGATE; + } + + private static void appendHexClassRange(StringBuilder out, long start, long end) { + out.append("\\x{") + .append(Long.toHexString(start).toUpperCase(java.util.Locale.ROOT)) + .append('}'); + if (start != end) { + out.append("-\\x{") + .append(Long.toHexString(end).toUpperCase(java.util.Locale.ROOT)) + .append('}'); + } + } + + private static String fixedWidthHexRange(int start, int end, int width) { + if (width <= 0) return ""; + int divisor = 1 << ((width - 1) * 4); + int startDigit = start / divisor; + int endDigit = end / divisor; + int startRemainder = start % divisor; + int endRemainder = end % divisor; + if (startDigit == endDigit) { + return hexDigit(startDigit) + + fixedWidthHexRange(startRemainder, endRemainder, width - 1); + } + + List alternatives = new ArrayList<>(); + alternatives.add(hexDigit(startDigit) + + fixedWidthHexRange(startRemainder, divisor - 1, width - 1)); + for (int digit = startDigit + 1; digit < endDigit; digit++) { + alternatives.add(hexDigit(digit) + fullHexSuffix(width - 1)); + } + alternatives.add(hexDigit(endDigit) + + fixedWidthHexRange(0, endRemainder, width - 1)); + return alternatives.size() == 1 ? alternatives.get(0) + : "(?:" + String.join("|", alternatives) + ")"; + } + + private static String fullHexSuffix(int width) { + if (width <= 0) return ""; + return width == 1 ? "[0-9A-F]" : "[0-9A-F]{" + width + "}"; + } + + private static char hexDigit(int value) { + return "0123456789ABCDEF".charAt(value); + } + + /** + * Keep an ordinary class alternative from starting inside the visible + * payload of an internal scalar marker during an unanchored search. Each + * lookbehind is fixed-width for Joni; an unsigned Perl UV has at most 16 + * hexadecimal digits. + */ + private static String internalScalarBoundaryGuard() { + StringBuilder guard = new StringBuilder("(?