Phase 36: make Joni the ordinary regex default - #1080
Merged
Conversation
Carry byte/Unicode backing through composed regex templates, recompilation, cloning, and nested callback templates so /d can select the correct Joni input variant without inferring encoding from decoded Java strings. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <noreply@openai.com>
Mark byte-backed Joni patterns explicitly, suppress Unicode multi-character case-fold expansion for them, and retain pinned Latin-1 simple-fold siblings. Add direct and Perl-facing regression coverage for literal and backreference sharp-s behavior. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <noreply@openai.com>
Resolve dotted U+ named-character sequences only in regex source while retaining Perl's scalar-string error. Report an unmatched character class before the valid sequence and attach the exact compile-site source location. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <codex@openai.com>
Add a system-Perl-validated matrix for ordinary constant patterns across captures, classes, assertions, modifiers, reuse, substitution, and match state. This freezes explicit Java and Joni behavior before changing the default route. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <codex@openai.com>
Route default and auto regex compilation through Joni while retaining explicit Java as a temporary differential baseline. Update the cache mode identity and the obsolete policy expectations to match the permanent production route. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <codex@openai.com>
Preserve ASCII byte-pattern provenance for reverse folds, restrict byte /d folding to ASCII, and retain direct Unicode property membership under /aa property closure. Add a compact system-Perl-derived matrix across modes and backreference forms. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <noreply@openai.com>
Avoid changing callback global-match state for byte-backed patterns that have no case-folding semantics, while retaining byte /d selection for /i patterns. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <noreply@openai.com>
Add a system-Perl-derived contract for undefined, empty, quantified, modifier, position, callback-result, exception, and invalid-source dynamic patterns. Render an invalid returned character class with Perl's marked diagnostic. Generated with Codex (https://openai.com/codex) Co-Authored-By: OpenAI Codex <codex@openai.com>
Keep a stable sequence head while expanding multi-character ASCII byte-pattern folds. ListNode.listAdd returns the appended tail, so assigning it back to the root discarded every preceding literal segment. Add direct exact, folded, and negative anchored coverage. Generated with Codex (https://openai.com/codex) Co-Authored-By: OpenAI Codex <codex@openai.com>
The bulk perl5/t import already restores re/pat.t byte-for-byte now that the historical workaround patch is retired. Remove the duplicate file row after verifying two-pass sync idempotence and the unpatched upstream Joni gates. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <codex@openai.com>
Align byte-pattern variant construction with selection. Unicode, ASCII, and ASCII-strict modes never select the ISO byte variant, so compiling it eagerly caused property and class fold failures before the intended matcher was used. Generated with Codex (https://openai.com/codex) Co-Authored-By: OpenAI Codex <codex@openai.com>
Do not build the default ISO byte matcher for named characters, wide numeric escapes, or literal wide code points. These constructs promote otherwise ASCII source to Unicode semantics and byte-variant compilation duplicated lexical translator calls or admitted excluded Turkic folds. Generated with Codex (https://openai.com/codex) Co-Authored-By: OpenAI Codex <codex@openai.com>
Derive effective byte provenance before constructing the regex cache key and storing the compiled pattern. Named characters, wide numeric escapes, and wide literal source now share one Unicode-backed cache entry between literal validation and runtime compilation, preserving one lexical charname expansion. Generated with Codex (https://openai.com/codex) Co-Authored-By: OpenAI Codex <codex@openai.com>
Translate Joni's generic undefined-group-option error only when its marker follows a literal escaped group token, preserving Perl's exact diagnostic and source marker across ordinary and forced-Joni execution. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <codex@openai.com>
Translate Joni's oversized repeat-range diagnostic to Perl's stable quantifier error family while preserving the engine-provided source marker. Add a compact system-Perl-validated parity matrix for exact, open-ended, and bounded oversized quantifiers. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <codex@openai.com>
Translate Joni's generic end-of-pattern parenthesis error for ordinary groups and place Perl's diagnostic marker at the unmatched opener. Keep specialized (?...) constructs on their dedicated diagnostic paths. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/dcase-fold variants only where Perl byte semantics require them/u,/a,/aa, or Unicode-promoting sourcepat.tmanifest rowU+named sequences, escaped group options, oversized quantifiers, and unmatched ordinary group openersValidation
make: all 17 tasks green at semantic head66f224b28(/tmp/make-phase36-v9-effective-byte-provenance.log)make: all 17 tasks green at1938c1672in 2m52s (/tmp/make-phase36-v9-pr-head.log)reg_mesg.t: 1754 to 1794/2613 on JVM and interpreter, 40 failures removed and zero introduced by the final diagnostic tranchepat.tis byte-identical to upstreamCompatibility
Explicit Java routing remains available only for differential validation. The
Phase 36 plan still requires removal of Java matching, the selector, and
matcher-semantic preprocessing after the remaining exact-artifact corpus and
release gates pass.
Generated with Codex (https://openai.com/codex)