Skip to content

feat(regex): complete native Joni migration - #1087

Open
fglock wants to merge 332 commits into
masterfrom
integrate/phase36-recursion-barrier
Open

feat(regex): complete native Joni migration#1087
fglock wants to merge 332 commits into
masterfrom
integrate/phase36-recursion-barrier

Conversation

@fglock

@fglock fglock commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

  • make the vendored Joni fork the sole production Perl regex matcher
  • move Perl grammar, backtracking-visible state, callbacks, Unicode semantics,
    diagnostics, and control verbs into native Joni/runtime integration
  • remove the Java matcher path, matcher-semantic preprocessing, obsolete import
    patches, and dead migration switches
  • add latest-Perl Unicode generators and strict PR-958 differential tooling
  • reconcile the current regex architecture, fork contract, and release plan

Incremental release checkpoint

The exact candidate is cb0926a5c on top of current master.

  • warning-free exact make in 5m13s, licensed Joni, all unit shards,
    packaging, and shadow JAR
  • planning 622-file A16 run: 677,871/695,187 assertions, 95 improved rows,
    and no isolated regex pass decrease from PR 958
  • non-regex overload decrease fixed: unchanged overload_fallback.t 4/4
    on JVM and interpreter
  • io/socket.t classified as sandbox-only: exact unsandboxed runner 25/25,
    zero errors/timeouts
  • raw Perl property resolution, escaped-class callbacks, byte NEL, and
    word-boundary whitespace semantics pass focused JVM/interpreter gates
  • regexp.t 2210/2210 on both backends for the byte-NEL tranche
  • uniprops10.t 20,070/20,070 on both backends; the two native boundary
    tranches recover 1,190 assertions with zero introductions
  • bounded speed* 59/59 and pat_psycho* 17/17 on JVM/interpreter and
    thread wrappers; five-run warmed ordinary-regex median improves versus parent
  • fresh user-acceptance 622-file run at exact cb0926a5c, strict PR-958
    comparison, and additional release checks
  • Ubuntu and Windows CI on this exact pushed head

The A16 planning-run dossier and immutable artifacts are retained under
../PerlOnJava/logs/test_20260821_080900_d3a4bb074_a16_*; they prevent repeated
root discovery but do not replace the fresh exact-head user-acceptance run.
Every subsequent semantic tranche has standard-Perl, dual-backend,
affected-corpus, and zero-introduction evidence.

Merge requires the fresh user run, strict comparison, additional release
checks, and CI to confirm the project's incremental rule: every valid PR-958
row is retained after documented platform/fixture normalization and at least
one row improves. It does not declare Phase 36 complete.

Work continuing after this checkpoint

  • close the ordinary native matcher checklist and remaining matcher-state root
  • add generated current-Perl InSC/InPC data (2,120 residual assertions)
  • close the frontend aggregate regex-binding warning and remaining diagnostics
  • remove the last extended-property source-policy scanner through native Joni
    context propagation
  • qualify and then remove regex JPERL_UNIMPLEMENTED=warn scaffolding after
    this PR merges
  • finish complete dual-backend acceptance, final docs/POD reconciliation, and
    final CI/release gates

fglock and others added 30 commits August 20, 2026 14:14
@fglock @codex
refactor(regex): remove Java matcher execution storage
1b64005 
Route RuntimeRegex matching, zero-width retry, substitutions, and empty-pattern
reuse through the selected Joni pattern. Delete the obsolete Java matcher adapter.

Generated with Codex
https://openai.com/codex

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
refactor(regex): retire Java regex compilation frontend
c33f119 
Compile and execute Perl regexes exclusively with Joni, remove obsolete Java
Pattern storage and preprocessor translation classes, and retain source policy
diagnostics through the native compilation path.

Generated with Codex
https://openai.com/codex

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs(regex): mark Java matcher frontend retired
9b00136 
Record the integrated native-only matcher storage and preprocessing cutover in
the history-free Phase 36 execution tracker.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
refactor(regex): remove obsolete Java pattern flags
77a6c91 
Drop the unused RegexFlags adapter and RuntimeRegex constants that existed only
for the retired Java matcher compilation path.

Generated with Codex
https://openai.com/codex

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs(regex): remove retired frontend references
b1dcec4 
Update source comments to describe native compilation ownership after deleting
the legacy regex preprocessor classes.

Generated with Codex
https://openai.com/codex

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs(regex): align final implementation workstreams
c28641b 
Replace stale Java-frontend and invalid fold targets with the current native
diagnostics, byte-fold, generated-name, and optimizer completion slices.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
feat(unicode): generate Perl scalar character names
116a035 
Generate a compact deterministic lookup from the current imported Name.pl so
standard named-character expansion recognizes Perl's canonical names and all
Unicode alias rows before falling back to ICU.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs: require immutable build worktrees
f140843 
Record the invalidated Phase 36 integration build and require builds and test
gates to drain before their checkout is mutated.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs(regex): mark generated scalar names complete
296b2c7 
Update the history-free execution tracker after integrating the deterministic
Perl scalar character-name and alias lookup.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
fix(regex): diagnose operand-less extended class operators
ee6756c 
Make the native Joni parser report Perl-compatible diagnostics when a binary
extended-character-class operator has no left operand, retaining the source
position after the offending operator. Add direct Joni and Perl reducer tests.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
fix(regex): fold Unicode casing property aliases
212e584 
Derive Upper and Lower fold policy from ICU's canonical property enums so all
aliases of the Unicode Uppercase and Lowercase properties share Perl's /i
semantics without special-casing individual spellings.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs(regex): align final tracker and ownership
85dec8c 
Name the live deferred-property owner after removal of the regex preprocessor,
record completion of Unicode casing-property alias folding, and distinguish
removed production Java routing from the disconnected policy parser retained
only for immutable compatibility tests.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
fix(regex): diagnose missing extended class operators
018f80d 
Report Perl-compatible diagnostics for adjacent extended-class operands and
misplaced parentheses while retaining legal nested groups. Add direct Joni and
system-Perl-validated reducer coverage.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs(regex): record final sole-Joni architecture
7d34656 
Replace obsolete Java matcher and preprocessing descriptions with the
as-implemented runtime/source-policy boundary. Document the complete callout
ABI, byte and Unicode encodings, current-Perl generated data policy, optimizer
and cache ownership, relocation notices, and remaining acceptance gates.

Generated with [Codex](https://openai.com/codex/)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs(regex): update native diagnostic position
5054f23 
Record the integrated second extended-class diagnostic family and its exact
zero-introduction reg_mesg.t improvement in the canonical Phase 36 plan.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
fix(regex): render overlong lookbehind diagnostics faithfully
99e887c 
Preserve Perl's m/pattern/ context for native Joni lookbehind-length errors
without adding a duplicate phrase or source marker. Add a system-Perl-validated
cross-backend reducer.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs(regex): update diagnostic parity position
2c88e30 
Record the integrated overlong-lookbehind renderer and the resulting exact
reg_mesg.t count on both execution backends.

Generated with [Codex](https://openai.com/codex/)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs(regex): qualify capability matrix evidence
06e54e3 
Keep the POD inventory result durable in the feature matrix and classify
literal/class folding plus case-insensitive backreference folding as partial
until the final integrated artifact proves those semantic boundaries.

Generated with [Codex](https://openai.com/codex/)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs(regex): retire superseded migration designs
3de516a 
Remove obsolete callback, differential, multi-engine, parser, preprocessing,
and Joni migration sketches. Point their inbound references at the canonical
Phase 36, implementation, and callout-contract documents.

Generated with [Codex](https://openai.com/codex/)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
refactor(regex): remove dead Java class adapter
a7ea5b6 
Delete the unreferenced helper that evaluated translated Java character classes
while constructing Joni source. Native extended classes no longer call it.

Generated with [Codex](https://openai.com/codex/)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
perf(regex): prune impossible dot-star retries in Joni
d09fc62 
Recognize exact bytecode immediately following generic ANYCHAR_STAR opcodes
and retain backtracking alternatives only at positions that can satisfy its
first byte. Keep template-backed and case-insensitive layouts on the general
path, with coverage for ASCII, multibyte UTF-8, lazy/greedy, and newline cases.

Generated with [Codex](https://openai.com/codex)

Co-Authored-By: Codex <noreply@openai.com>
@fglock @codex
perf(regex): reuse direct scalar input encodings
827d138 
Cache Joni byte/character offset maps by weak RuntimeScalar identity only for
direct immutable STRING and BYTE_STRING storage whose value is the rendered
input object. Keep byte and Unicode modes separate, invalidate on scalar state
changes, and fall back for tied, overloaded, indirect, dual, or other rendered
values so changing stringification cannot return stale offsets.

Generated with [Codex](https://openai.com/codex)

Co-Authored-By: Codex <noreply@openai.com>
@fglock @codex
perf(regex): avoid per-match ThreadLocal churn in Joni
c25249f 
Store matcher execution depth in one mutable thread-local holder instead of
boxing, setting, and removing an Integer for every match. Preserve independent
stacks for same-thread callback and dynamic matcher reentry, with cross-thread
and long global-match regression coverage validated against system Perl.

Generated with [Codex](https://openai.com/codex)

Co-Authored-By: Codex <noreply@openai.com>
@fglock @codex
feat(tools): add fail-closed Phase 36 acceptance driver
0f1ab03 
Compose the current-checkout ledger, bounded JVM and interpreter runner legs,
PR-958-normalized comparisons, and exact packaging verifier into a manifest
with retained artifact hashes. Cover prepare-only list-form orchestration with
fake tools and temporary fixtures.

Generated with [Codex](https://openai.com/codex/)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
fix(tools): verify packaged Joni provenance and SBOM edge
151c9b6 
Require relocated unique classes, byte-identical Joni notices, exact unique
Joni/JCodings CycloneDX identities, and the declared dependency relation.
Cover synthetic green and fail-closed packaging fixtures.

Generated with [Codex](https://openai.com/codex/)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
fix(tools): bind regex acceptance to source and runner identity
c42a4b1 
Require clean invariant tracked source state and a bounded jperl version SHA
that matches the verified source. Retain and hash the version log, and extend
the packaging fixture failure matrix.

Generated with [Codex](https://openai.com/codex/)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
fix(tools): bound regex acceptance identity checks
b8b3a11 
Hard-bound the jperl version probe with portable exact-child cleanup and a
distinct timeout status. Validate the wrapper as executable and exercise every
duplicate packaging and missing vendored-component failure family.

Generated with [Codex](https://openai.com/codex/)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs(regex): advance final acceptance position
18ac9f7 
Record the fail-closed acceptance and packaging boundary, remove completed work
from the ordered queue, and mark the completed conditions/state and migration-
scaffolding phases while retaining the active fold, diagnostics, and release
gates.

Generated with [Codex](https://openai.com/codex/)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
fix(regex): diagnose unterminated conditionals
f63ce63 
Translate the native parser's generic unmatched-parenthesis error to Perl's
conditional-specific termination diagnostic. Add a system-Perl-validated
cross-backend reducer.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
fix(regex): diagnose unsupported group introducers
6b66d17 
Use Perl-compatible diagnostics for unsupported (?@...) and unrecognized
(?;...) group introducers. Add a system-Perl-validated cross-backend reducer.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
fglock and others added 28 commits August 21, 2026 07:55
@fglock @codex
fix(regex): keep VLB warnings default-on
1e15462 
Match Perl's default warning policy for experimental variable-length
lookbehind captures while retaining lexical category suppression and fatal
warning handling.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fix: recognize digit-suffix user properties
d3a4bb0 
Treat Is*/In* names with digit-leading suffixes as user-defined property
candidates so extended sets report Perl's specific unresolved-property
diagnostic.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
docs(regex): advance current green ownership
e32b19b 
Record the exact combined v7 gate, close the integrated property and warning
roots, and assign the next byte-class, warning, interpolation, and private
acceptance slices.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
docs(regex): replace history with live execution plan
26d959b 
Compress the Phase 36 release gate into current acceptance criteria,
ownership, and ordered next steps. Keep implementation history in commits.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fix(regex): retain single-byte portions of code ranges
b4e56a7 
Store the byte-domain portion of Joni character-class code ranges in the
single-byte bitset when compiling under a single-byte encoding. This restores
Perl's negated POSIX ASCII semantics for Latin-1 byte subjects and closes the
three corresponding regexp corpus assertions.

Add latest-Perl and licensed direct-Joni regression coverage.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fix(regex): warn for mixed strict range endpoints
f1af949 
Emit Perl's Unicode endpoint-provenance warning for accepted ordinary
character-class ranges under re strict, matching the existing extended-class
policy.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fix(regex): avoid duplicate strict class warning
85861fc 
Let the ordinary strict lexer remain the sole owner of the unescaped closing
bracket warning after malformed spaced POSIX-class diagnostics.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fixup! fix(regex): warn for mixed strict range endpoints
44542b6 
Limit mixed-provenance warnings to raw numeric byte escapes and retain quiet
Unicode-equivalent controls discovered by the pristine private corpus.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
docs(regex): record native migration and acceptance gates
bf7734b 
Capture the complete 622-file release contract, current worker ownership,
completed import cleanup, and the remaining raw-property scanner migration.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
docs(regex): assign native property scanner tranche
c51a50b 
Give P4 the complete raw-property resolver and adapter-scanner removal outcome.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fix: retain utf8 pragma in interpreter source scope
18b9a01 
Keep the public utf8 source-decoding hint on the enclosing parser scope when
an interpreter-backed synthetic import frame restores its local pragma state.

Generated with Codex (https://openai.com/codex)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
docs(regex): refresh active release barrier
d5dd8c0 
Record the validated integration head, retained complete-corpus evidence, and
the current outcome-sized worker ownership for the remaining parity roots.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fix: parse recursively scoped extended-class interpolation
2690b70 
Allow Perl extended character classes to interpolate an eventual extended
class leaf through multiple nested scoped modifier groups while restoring
each group's option state and preserving invalid-leaf diagnostics.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
docs(regex): record complete acceptance disposition
9480312 
Retain the reusable 622-file evidence, identify the two genuine non-regex
release blockers, and route the three largest regex residual families.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fix(runtime): increment plain overload copy results
cbee747 
When an overloaded copy constructor returns a plain scalar and neither an
explicit increment nor addition overload handles the mutation, apply Perl's
native increment to the copied value. Preserve the corresponding postincrement
return value.

This restores the latest perl5 overload_fallback.t autogeneration assertion.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
docs(regex): narrow the next release blocker
a40cb91 
Record the validated overload fallback correction and make io/socket.t the sole
remaining non-regex blocker for the next incremental PR checkpoint.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fix(regex): keep Perl word-boundary newline runs intact
16dd30f 
Treat adjacent CR, LF, and Newline word-break properties as one Perl word
segment while preserving boundaries at the run edges. Add direct Joni and
system-Perl-backed runtime coverage for positive and negated boundaries,
start/end positions, and repeated zero-width scans.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
docs(regex): record the word-boundary tranche
f8430cc 
Advance the integration barrier after the newline-run boundary correction and
route the remaining generated failures to ignored-sequence state handling.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
docs(regex): classify socket decreases as sandbox effects
9935dfd 
Record the exact unsandboxed 25/25 runner result and focused JVM/interpreter
socket-option evidence, removing the last reproduced non-regex release blocker.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fix(regex): keep property wildcard warnings default-on
5111806 
Snapshot lexical warning policy across retained Unicode property wildcard
diagnostics so Perl's feature notice remains default-on while the positioned
diagnostic still follows use warnings, suppression, and fatality.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fix(regex): preserve escaped class opener parity
2117fe0 
Track backslash parity outside regex character classes so an escaped opening
bracket cannot hide literal callback source from compile-time extraction.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fixup! fix(regex): keep property wildcard warnings default-on
986b83b 
Retain runtime, call-site, and active lexical warning snapshots before the
first warning callback so explicit and inherited eval warning scopes both
govern the positioned follow-up diagnostic.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
fix(regex): resolve Perl properties through Joni hook
96aa0b7 
Compile original property source through the runtime-neutral Joni resolver
instead of expanding built-in properties into generated class text. Preserve
Perl alias, wildcard, boolean-complement, user-property, and extended-class
policy across byte and Unicode modes.

Generated with [Codex](https://openai.com/codex/)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
fix: recognize byte NEL as a line break
366e1e0 
Include U+0085 in Perl line-break classes for single-byte encodings while
keeping LS and PS Unicode-only and preserving CRLF and vertical-space behavior.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
docs(regex): advance the checkpoint composition
b693f56 
Record the green native property and warning composition, the queued byte-NEL
tranche, and the active post-checkpoint workstreams.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
docs: reconcile regex and Joni architecture
01740cc 
Document the sole-Joni runtime boundary, trusted callback slot provenance,
Unicode data ownership, remaining source-policy scans, and fork validation
contracts against the integrated implementation.

Generated with [Codex](https://openai.com/codex/)

Co-Authored-By: Codex <codex@openai.com>
@fglock @codex
fix(regex): preserve Perl word-boundary whitespace spans
702bb28 
Model Perl's tailored horizontal-space word-boundary class alongside CR,
LF, and Newline runs. Keep horizontal spans joined to newline properties,
and bind a trailing horizontal space to Extend or Format while preserving
Perl's distinct ZWJ and ExtendNumLet behavior.

Add direct Joni and system-Perl-backed coverage for ignored sequences,
Unicode spaces, RI and emoji state, negated scans, and backtracking.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock @codex
docs(regex): advance the checkpoint release gate
cb0926a 
Record the integrated word-boundary closure, current worker ownership, and the
single exact combined-build barrier for PR 1087.

Generated with [OpenAI Codex](https://openai.com/codex/)

Co-Authored-By: OpenAI Codex <codex@openai.com>
@fglock
fglock marked this pull request as ready for review August 21, 2026 08:17
@fglock Codex
chore: sync bundled Perl libraries
5a1e0c1 
Import the user-run perl5 sync results, including upstream module version
bumps and Perl 5.45.1 through 5.45.3 POD documentation.

Generated with Codex (https://openai.com/codex/)

Co-Authored-By: Codex <openai-codex@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@fglock