From 18b9c4db2f92ac08e78dd4b1368972bd85760dc8 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:07:41 +0200 Subject: [PATCH 01/60] test(regex): report full forced backend differential (#1006) Squashed integration of PR #1006 at e1d96ef31. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- .../phase36-regex-differential-20260817.md | 185 ++++++++++++++++++ dev/tools/README.md | 7 +- dev/tools/compare_test_results.pl | 57 +++++- dev/tools/tests/compare_test_results.t | 23 +++ 4 files changed, 267 insertions(+), 5 deletions(-) create mode 100644 dev/design/phase36-regex-differential-20260817.md diff --git a/dev/design/phase36-regex-differential-20260817.md b/dev/design/phase36-regex-differential-20260817.md new file mode 100644 index 0000000000..e4b47f09cc --- /dev/null +++ b/dev/design/phase36-regex-differential-20260817.md @@ -0,0 +1,185 @@ +# Phase 36 Forced-Backend Regex Differential — 2026-08-17 + +## Scope + +This report records the 80-file `perl5_t/t/re` corpus with the temporary Java +and Joni matcher selector on both PerlOnJava execution backends. The corpus +uses the clean pre-PR-1002 binary built from commit `ba85bdc05`; its SHA-256 is: + +```text +04be3d3d52348dbda4de1ff187df5cd9d788a274ab14bbb5170bafe879cd12cf +``` + +The historical comparison is PR 958's captured runner log: + +```text +../PerlOnJava/logs/test_20260815_080000_958.log +``` + +That log contains all 80 regex files and records 50,273/94,771 passing +assertions. Complete logs, runner JSON, and normalized comparison JSON remain +outside Git under `/tmp/phase36-regex-*`. + +PR 1002 merged and PR 1005's package-localization fix was prepared after this +binary was frozen. In particular, PR 1005 closes `pat_advanced.t` tests +922–933. Those changes are intentionally not mixed into this same-commit +matrix; a later verification rerun must use one rebuilt current-master binary +for all four legs. + +## Method + +Each matrix leg inherits one matcher selection and, for interpreter legs, +global interpreter selection. `perl_test_runner.pl` owns a hard timeout and +process group for every `jperl` child. Full output is captured to a log and the +runner writes per-file JSON. The displayed 300-second value is the base limit; +the runner's configured heavy-file overrides raise `pat_psycho*` and other +designated heavy fixtures to 600 seconds. + +```text +JPERL_REGEX_BACKEND=java perl dev/tools/perl_test_runner.pl \ + --timeout 300 --jobs 5 --output /tmp/phase36-regex-java-jvm.json perl5_t/t/re + +JPERL_REGEX_BACKEND=joni perl dev/tools/perl_test_runner.pl \ + --timeout 300 --jobs 5 --output /tmp/phase36-regex-joni-jvm.json perl5_t/t/re + +JPERL_REGEX_BACKEND=java JPERL_INTERPRETER=1 \ + perl dev/tools/perl_test_runner.pl --timeout 300 --jobs 6 \ + --output /tmp/phase36-regex-java-interpreter.json perl5_t/t/re + +JPERL_REGEX_BACKEND=joni JPERL_INTERPRETER=1 \ + perl dev/tools/perl_test_runner.pl --timeout 300 --jobs 6 \ + --output /tmp/phase36-regex-joni-interpreter.json perl5_t/t/re +``` + +The comparison gate now reports execution errors, incomplete files, and +zero-TAP results explicitly. `--expected-files 80 --fail-on-invalid` rejects an +incomplete corpus independently of the existing per-file pass-count regression +gate. + +## Matrix Summary + +| Matcher | Execution backend | Passing / planned | Delta vs PR 958 | Timeouts | Status | +|---|---|---:|---:|---:|---| +| Java | JVM | 49,923 / 94,823 | -350 / +52 | 0 | complete | +| Joni | JVM | 32,479 / 77,612 | -17,794 / -17,159 | 10 | complete | +| Java | interpreter | 50,021 / 94,823 | -252 / +52 | 0 | complete | +| Joni | interpreter | 32,483 / 77,612 | -17,790 / -17,159 | 10 | complete | + +## Complete Java/JVM Comparison + +The forced-Java/JVM leg covered all 80 files. It has four per-file pass-count +regressions against PR 958: + +| File | PR 958 | Candidate | Delta | Initial classification | +|---|---:|---:|---:|---| +| `pat.t` | 1099/1302 | 239/1302 | -860 | shared source-provenance abort | +| `pat_thr.t` | 1099/1302 | 239/1302 | -860 | same direct failure in thread wrapper | +| `reg_mesg.t` | 1692/2525 | 1664/2521 | -28 | diagnostics and plan change | +| `regexp_normal.t` | 2193/2210 | 2175/2210 | -18 | matcher semantics | + +`pat{,_thr}.t` stops immediately after test 239 with: + +```text +Eval-group not allowed at runtime, use re 'eval' +``` + +This is not an optimizer/debug transcript difference or a Java/Joni routing +difference. The forced-Joni leg stops at the same point. The failing literal +uses single-quote regex delimiters, `m'a(?{ ... })b'`; initial reduction shows +that this parse path produces a plain pattern string and loses trusted literal +callback provenance before matcher selection. + +The net aggregate loss masks substantial progress since PR 958. In particular, +`pat_re_eval{,_thr}.t` improves from 0/555 to 423/555 each, +`pat_advanced{,_thr}.t` improves from 1376/1687 to 1511/1687 each, +`reg_eval_scope.t` reaches 49/49, and `rxcode.t` reaches 40/42. + +## Complete Joni/JVM Comparison + +The forced-Joni leg covers all 80 files but falls to 32,479/77,612: 17,794 +fewer passing assertions and 17,159 fewer planned assertions than PR 958. It +contains 27 per-file pass-count regressions, 33 execution issues, and 29 +zero-TAP files. Ten files reach their runner-owned bounds with zero TAP: + +- 300 seconds: `regex_sets_compat.t`, `regexp.t`, `regexp_noamp.t`, + `regexp_notrie.t`, `regexp_qr.t`, `regexp_qr_embed.t`, and + `regexp_trielist.t`. +- 600 seconds: `pat_psycho.t`, `pat_psycho_thr.t`, and + `regexp_qr_embed_thr.t`. + +The cluster is matcher-specific: the same files complete in the forced-Java +leg. Reduction must determine whether each stall occurs during Joni compile or +search before any broad implementation change. + +Completed Joni/JVM files also expose semantic gaps independent of timeout: + +- `pat{,_thr}.t`: 239/1302 each, sharing the matcher-independent literal + callback provenance abort described above. +- `pat_advanced{,_thr}.t`: 1271/1687 each, versus forced Java 1511/1687. +- `reg_mesg.t`: 1364/2486, versus forced Java 1664/2521. +- `reg_posixcc.t`: 2052/2560, versus forced Java 2560/2560. +- `regexp_normal.t`: 2175/2210 on both matcher selections. +- `regexp_unicode_prop.t`: 1039/1110; its thread wrapper is 1040/1110. + +`speed.t` exits incomplete at 1/59 after 451 seconds and `speed_thr.t` exits +incomplete at 1/59 after 301 seconds. These are internal-watchdog/incomplete +results rather than runner timeouts, but still lose 25 passing assertions per +file against PR 958. + +## Complete Java JVM/Interpreter Comparison + +The forced-Java interpreter leg covers all 80 files at 50,021/94,823. It has +the same zero runner timeouts and planned count as forced-Java/JVM, while +gaining 98 passing assertions in aggregate. That aggregate gain is entirely +driven by `alpha_assertions.t` (2188/2320 on JVM and 2293/2320 on interpreter, ++105); four interpreter-specific losses remain: + +| File | JVM | Interpreter | Delta | +|---|---:|---:|---:| +| `reg_eval_scope.t` | 49/49 | 45/49 | -4 | +| `pat_advanced.t` | 1511/1687 | 1510/1687 | -1 | +| `pat_advanced_thr.t` | 1511/1687 | 1510/1687 | -1 | +| `qr.t` | 4/4 | 3/4 | -1 | + +Against PR 958, the interpreter leg has five pass-count regressions: the same +four Java/JVM corpus regressions plus `qr.t` at -1. These execution-backend +differences are shared runtime/compiler parity gaps, not Java-vs-Joni matcher +differences. + +## Complete Joni JVM/Interpreter Comparison + +The forced-Joni interpreter leg covers all 80 files at 32,483/77,612. Its ten +timeouts, 33 execution issues, 29 zero-TAP files, and planned count are +identical to Joni/JVM. It gains four passing assertions in aggregate, again +masking execution-backend differences: + +| File | JVM | Interpreter | Delta | +|---|---:|---:|---:| +| `reg_eval_scope.t` | 49/49 | 45/49 | -4 | +| `qr.t` | 4/4 | 3/4 | -1 | +| `alpha_assertions.t` | 2099/2320 | 2108/2320 | +9 | + +On the interpreter execution backend, selecting Joni instead of Java changes +50,021/94,823 to 32,483/77,612: 17,538 fewer passing and 17,211 fewer planned +assertions, with 29 regressing and three improving files. The loss categories +match the JVM engine differential, so the primary blocker is matcher selection +rather than execution-backend compilation. + +## Classification and Follow-up + +- Source policy: fix the matcher-independent single-quote literal callback + provenance abort after this report PR, with a focused standard-Perl oracle. +- Matcher semantics: separate the shared `regexp_normal.t` loss from + Joni-only `pat_advanced.t` and POSIX/property losses. +- Diagnostics: treat `reg_mesg.t` independently from matching behavior. +- Timeout behavior: reduce the Joni timeout cluster under individual hard + timeouts; do not hide it through Java fallback. +- Optimizer/debug transcript: none of the regressions above are currently + classified as transcript-only exclusions. + +The Phase 1 exit criterion is not met. Joni fails to retain every assertion +that passes under Java, introduces ten matcher-specific timeouts on both +execution backends, and exposes independent semantic and diagnostic losses. +The four relevant pairwise comparisons (Java/Joni within each execution +backend and JVM/interpreter within each matcher selection) are complete and +retained under `/tmp/phase36-compare-*`. diff --git a/dev/tools/README.md b/dev/tools/README.md index 7a494a1098..835e318061 100644 --- a/dev/tools/README.md +++ b/dev/tools/README.md @@ -43,10 +43,13 @@ parallelism without making their watchdogs contend with `pat*`. The baseline may be either runner JSON or a captured historical runner log. Use `--fail-on-regression` for release gates and `--output` to retain a -machine-readable per-file comparison: +machine-readable per-file comparison. Corpus gates can additionally use +`--fail-on-invalid --expected-files N` to reject missing files, execution +errors, timeouts, incomplete runs, and zero-TAP results: ```bash -perl dev/tools/compare_test_results.pl --fail-on-regression \ +perl dev/tools/compare_test_results.pl --fail-on-regression --fail-on-invalid \ + --expected-files 80 \ --path-prefix perl5_t/t/re \ --output /tmp/regex-comparison.json \ ../PerlOnJava/logs/test_20260815_080000_958.log /tmp/regex.json diff --git a/dev/tools/compare_test_results.pl b/dev/tools/compare_test_results.pl index c6653732ea..cd911a302b 100644 --- a/dev/tools/compare_test_results.pl +++ b/dev/tools/compare_test_results.pl @@ -7,11 +7,15 @@ use Encode qw(FB_DEFAULT decode); my $fail_on_regression = 0; +my $fail_on_invalid = 0; +my $expected_files; my $output_file; my $path_prefix; my $help = 0; GetOptions( 'fail-on-regression!' => \$fail_on_regression, + 'fail-on-invalid!' => \$fail_on_invalid, + 'expected-files=i' => \$expected_files, 'output=s' => \$output_file, 'path-prefix=s' => \$path_prefix, 'help' => \$help, @@ -30,12 +34,14 @@ $candidate = filter_results($candidate, $path_prefix); } my $comparison = compare_results($baseline, $candidate); +$comparison->{expected_files} = $expected_files if defined $expected_files; print_report($baseline_file, $candidate_file, $comparison); save_report($output_file, $baseline_file, $candidate_file, $comparison) if defined $output_file; exit 1 if $fail_on_regression && @{$comparison->{regressions}}; +exit 1 if $fail_on_invalid && has_invalid_candidate($comparison, $expected_files); exit 0; sub usage { @@ -48,6 +54,9 @@ sub usage { Options: --fail-on-regression Exit nonzero when any candidate file loses passing tests + --fail-on-invalid Exit nonzero for missing files, execution failures, + zero-TAP results, or an --expected-files mismatch + --expected-files NUM Require exactly NUM candidate files --output FILE Save the normalized comparison as JSON --path-prefix PATH Compare only files below this canonical path --help Show this help @@ -118,7 +127,8 @@ sub filter_results { sub compare_results { my ($baseline, $candidate) = @_; - my (@regressions, @improvements, @plan_changes, @missing, @added); + my (@regressions, @improvements, @plan_changes, @missing, @added, + @execution_issues, @zero_tap); my ($baseline_ok, $candidate_ok, $baseline_total, $candidate_total) = (0, 0, 0, 0); $baseline_ok += $_->{ok} for values %$baseline; @@ -127,6 +137,19 @@ sub compare_results { $candidate_total += $_->{total} for values %$candidate; my %all = map { $_ => 1 } (keys %$baseline, keys %$candidate); + for my $file (sort keys %$candidate) { + my $result = $candidate->{$file}; + push @execution_issues, { + file => $file, + status => $result->{status}, + ok => $result->{ok}, + total => $result->{total}, + } if $result->{status} =~ /\A(?:error|timeout|incomplete)\z/; + push @zero_tap, { + file => $file, + status => $result->{status}, + } if $result->{total} == 0; + } for my $file (sort keys %all) { my ($before, $after) = ($baseline->{$file}, $candidate->{$file}); if (!$after) { @@ -165,6 +188,8 @@ sub compare_results { plan_changes => \@plan_changes, missing_files => \@missing, added_files => \@added, + execution_issues => \@execution_issues, + zero_tap => \@zero_tap, }; } @@ -175,10 +200,16 @@ sub print_report { print "Candidate: $candidate_file\n"; printf "Passing assertions: %d/%d -> %d/%d (%+d passing, %+d planned)\n", @{$summary}{qw(baseline_ok baseline_total candidate_ok candidate_total delta_ok delta_total)}; - printf "Files: %d -> %d; regressions=%d improvements=%d missing=%d added=%d\n", + printf "Files: %d -> %d; regressions=%d improvements=%d missing=%d added=%d execution-issues=%d zero-TAP=%d\n", $summary->{baseline_files}, $summary->{candidate_files}, scalar(@{$comparison->{regressions}}), scalar(@{$comparison->{improvements}}), - scalar(@{$comparison->{missing_files}}), scalar(@{$comparison->{added_files}}); + scalar(@{$comparison->{missing_files}}), scalar(@{$comparison->{added_files}}), + scalar(@{$comparison->{execution_issues}}), scalar(@{$comparison->{zero_tap}}); + if (defined $comparison->{expected_files} + && $summary->{candidate_files} != $comparison->{expected_files}) { + printf "EXPECTED FILE COUNT MISMATCH: expected %d, found %d\n", + $comparison->{expected_files}, $summary->{candidate_files}; + } print_entries('REGRESSIONS', $comparison->{regressions}); print_entries('IMPROVEMENTS', $comparison->{improvements}); @@ -191,6 +222,26 @@ sub print_report { print "\nADDED FILES\n"; print " $_->{file}\n" for @{$comparison->{added_files}}; } + if (@{$comparison->{execution_issues}}) { + print "\nEXECUTION ISSUES\n"; + printf " %s: status=%s %d/%d\n", + @{$_}{qw(file status ok total)} for @{$comparison->{execution_issues}}; + } + if (@{$comparison->{zero_tap}}) { + print "\nZERO TAP\n"; + printf " %s: status=%s\n", @{$_}{qw(file status)} + for @{$comparison->{zero_tap}}; + } +} + +sub has_invalid_candidate { + my ($comparison, $required_files) = @_; + return 1 if @{$comparison->{missing_files}}; + return 1 if @{$comparison->{execution_issues}}; + return 1 if @{$comparison->{zero_tap}}; + return 1 if defined($required_files) + && $comparison->{summary}{candidate_files} != $required_files; + return 0; } sub print_entries { diff --git a/dev/tools/tests/compare_test_results.t b/dev/tools/tests/compare_test_results.t index 74126100d4..262826d9f0 100644 --- a/dev/tools/tests/compare_test_results.t +++ b/dev/tools/tests/compare_test_results.t @@ -43,6 +43,29 @@ my $document = JSON::PP->new->utf8->decode(do { local $/; <$fh> }); close $fh; is($document->{summary}{delta_ok}, 1, 'JSON report preserves aggregate delta'); is(scalar @{$document->{regressions}}, 1, 'JSON report preserves regressions'); +is_deeply($document->{execution_issues}, [], 'valid candidate has no execution issues'); +is_deeply($document->{zero_tap}, [], 'valid candidate has no zero-TAP results'); + +my $invalid = File::Spec->catfile($temporary, 'invalid.json'); +write_file($invalid, JSON::PP->new->pretty->encode({ + results => { + 'perl5_t/t/re/a.t' => { ok_count => 2, total_tests => 4, status => 'fail' }, + 'perl5_t/t/re/b.t' => { ok_count => 0, total_tests => 0, status => 'timeout' }, + }, +})); + +@command = ($^X, $tool, '--fail-on-invalid', '--expected-files', 80, + '--path-prefix', 'perl5_t/t/re', $baseline, $invalid); +$output = qx{@command 2>&1}; +is($? >> 8, 1, 'invalid-result mode exits nonzero'); +like($output, qr/execution-issues=1 zero-TAP=1/, + 'summary counts execution and zero-TAP issues'); +like($output, qr/EXPECTED FILE COUNT MISMATCH: expected 80, found 2/, + 'expected file-count mismatch is explicit'); +like($output, qr/EXECUTION ISSUES.*b\.t: status=timeout 0\/0/s, + 'timeout is reported as an execution issue'); +like($output, qr/ZERO TAP.*b\.t: status=timeout/s, + 'zero-TAP result is reported explicitly'); done_testing; From 26b483f09748427693ed116e5f8951aa59e79341 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:07:41 +0200 Subject: [PATCH 02/60] feat(regex): complete Phase 36 regex control semantics (#1005) Squashed integration of PR #1005 at 2bb3ebc4e. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 113 +++++++++++++++--- dev/import-perl5/config.yaml | 1 - dev/import-perl5/patches/pat.t.patch | 109 ----------------- .../backend/bytecode/BytecodeCompiler.java | 13 ++ .../runtime/operators/PerlUtfString.java | 22 ++++ .../runtime/operators/StringOperators.java | 23 +++- .../runtime/regex/JoniRegexPattern.java | 65 ++++++++-- .../regex/RegexPreprocessorHelper.java | 3 +- .../runtime/regex/RuntimeRegex.java | 78 ++++++++---- .../runtime/regex/RuntimeRegexTemplate.java | 61 +++++++++- .../runtime/regex/UnicodeResolver.java | 18 ++- .../runtimetypes/DynamicVariableManager.java | 13 ++ .../runtimetypes/GlobalRuntimeScalar.java | 15 +++ .../runtimetypes/RuntimePosLvalue.java | 20 ++++ .../unit/our_package_redeclaration.t | 23 ++++ .../unit/regex/accept_nested_capture_close.t | 37 ++++++ .../resources/unit/regex/callback_pos_state.t | 64 ++++++++++ .../regex/dynamic_code_array_interpolation.t | 83 +++++++++++++ .../invalid_unicode_property_diagnostics.t | 42 +++++++ .../unit/regex/last_successful_pattern.t | 52 ++++++++ .../unit/regex/reference_stringification.t | 16 +++ .../unit/regex/runtime_control_verb_package.t | 35 ++++++ .../runtime_dynamic_control_boundaries.t | 33 +++++ .../unit/regex/unicode_pos_offsets.t | 32 +++++ .../joni/src/org/joni/ByteCodeMachine.java | 47 ++++++++ third_party/joni/src/org/joni/Matcher.java | 23 ++++ .../joni/test/org/joni/test/TestCallout.java | 31 +++++ 27 files changed, 895 insertions(+), 177 deletions(-) delete mode 100644 dev/import-perl5/patches/pat.t.patch create mode 100644 src/test/resources/unit/our_package_redeclaration.t create mode 100644 src/test/resources/unit/regex/accept_nested_capture_close.t create mode 100644 src/test/resources/unit/regex/callback_pos_state.t create mode 100644 src/test/resources/unit/regex/dynamic_code_array_interpolation.t create mode 100644 src/test/resources/unit/regex/invalid_unicode_property_diagnostics.t create mode 100644 src/test/resources/unit/regex/last_successful_pattern.t create mode 100644 src/test/resources/unit/regex/reference_stringification.t create mode 100644 src/test/resources/unit/regex/runtime_control_verb_package.t create mode 100644 src/test/resources/unit/regex/runtime_dynamic_control_boundaries.t create mode 100644 src/test/resources/unit/regex/unicode_pos_offsets.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 654cf51cbe..e08971e59d 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -222,7 +222,14 @@ compatibility contract. ## Progress Tracking -### Current Status: Joni default; Phase 4 semantic gate complete at 550/555 +### Current Status: Joni default; Phase 1 remediation and Phase 3 Unicode active + +Invalid Unicode properties now remain fatal under +`JPERL_UNIMPLEMENTED=warn`, and only exact `Is`/`In` final-name prefixes enter +Perl's user-property callback path. A 39-assertion standard-Perl oracle passes +on JVM and interpreter; the unchanged `regexp_unicode_prop.t` runner improves +from 1,039/1,110 to 1,054/1,110, leaving built-in aliases and six +user-property-definition diagnostic assertions as separate work. Executable callback source and literal trailing `/x` comments survive canonical regex-object stringification on both execution backends. Recursive Joni call @@ -250,6 +257,23 @@ one-level failed callback state. The focused `pat_re_eval.t` gate executes all 555 assertions with 550 semantic assertions passing on both execution backends; the remaining five inspect Perl's optimizer/debug transcript. +The forced-backend differential is running in the parallel PerlOnJava3 +checkout. Its completed forced-Java/JVM leg covers all 80 files at +49,923/94,823 versus PR 958's 50,273/94,771. The apparent aggregate regression +is dominated by `pat{,_thr}.t` aborting after test 239 on a runtime eval-group +policy error; that source-policy slice is assigned independently. The completed +forced-Joni/JVM leg is 32,479/77,612, with ten bounded timeout files. Its +largest completed losses against forced Java are `reg_posixcc.t` (-508), +`reg_mesg.t` (-300), both `pat_advanced` variants (-240 each), +`alpha_assertions.t` (-89), and `regex_sets.t` (-84). The completed +forced-Java/interpreter leg covers all 80 files at 50,021/94,823 with no runner +timeouts, 98 more passing assertions than forced-Java/JVM, and an identical +plan. The completed forced-Joni/interpreter leg is 32,483/77,612 with the same +ten timeouts and planned count as Joni/JVM. The final same-binary report is +complete in `dev/design/phase36-regex-differential-20260817.md`; Phase 1's exit +criterion is not met because Joni loses Java-passing assertions and introduces +matcher-specific timeouts on both execution backends. + ### Completed Phases - [x] Phase 0: Reproducible differential baseline (2026-08-17) @@ -265,16 +289,67 @@ the remaining five inspect Perl's optimizer/debug transcript. - [x] Added the temporary backend selector and made Joni the default. - [x] Routed ordinary matching, substitution, and split through the selected backend without per-operation fallback. - - [ ] Re-run the complete forced-backend matrix and prove the exit criterion. -- [ ] Phase 2: Conditions and backtracking-visible state (in progress) + - [x] Completed the forced-Java/JVM 80-file leg and identified the + `pat{,_thr}.t` test-239 source-policy abort as the leading regression. + - [x] Completed the four-leg forced-backend matrix and published its + classification. The exit criterion is explicitly not met; timeout and + semantic remediation remain Phase 1 work. + - [x] Reduced the forced-Joni zero-pass surface to seven shared causes: + catastrophic backtracking, quadratic matcher reconstruction, absent + generated Unicode fixtures, regex-set preprocessing, unsupported compiler + introspection, regexp-object propagation, and three assertion-level + environment/runtime failures. + - [x] Moved immutable Joni UTF-8 input and offset maps out of the scalar + `/g` hot loop. The focused million-match oracle completes in 1.07 seconds + on JVM and 1.41 seconds on interpreter (PR #1008), with exact map and + supplementary-character capture-boundary coverage. +- [x] Phase 2: Conditions and backtracking-visible state (2026-08-17) - [x] Implemented executable callback conditions, control verbs including `(*MARK:NAME)`, and callback-visible recursive capture state in Joni. - [x] Closed runtime callback capture ownership at final scope teardown. - [x] Closed failed-path `$^N` and `$+` restoration through recursive callback unwind. + - [x] Added direct active-localization lookup for runtime control variables; + dynamic `PRUNE`, `SKIP`, and `COMMIT` update package `$REGERROR` without + mutating non-localized `$REGERROR`/`$REGMARK` variables on either backend. + - [x] Propagated `PRUNE`, `SKIP`, `COMMIT`, and `THEN` cuts and search-control + requests from nested `(??{...})` matcher programs. A 9-assertion + standard-Perl oracle passes on JVM and interpreter, and `pat_advanced.t` + test 891 now observes 3 callback executions instead of 9. + - [x] Refreshed the package alias stored for a reused `our` symbol when a + later declaration changes package. The focused package oracle passes on + system Perl, JVM, and interpreter, and `pat_advanced.t` tests 922-933 pass + on both execution backends without a regex-adapter workaround. + - [x] Exposed the actual match subject as callback `$_`, the provisional + callout offset through `pos`, and the in-progress match span through `$&` + plus the pre-match and post-match variables. Callback-bearing substitution + recompilation now preserves trusted callout markers. The 24-assertion + upstream `pos inside (?{})` block + passes on system Perl, JVM, and interpreter; `subst_amp.t` remains 13/13 + on both execution backends. + - [x] Removed the obsolete nested `(*ACCEPT)` and callback-`pos` workarounds + from `pat.t.patch` and resynchronized those original Perl 5.44 assertions. + - [x] Verified reference stringification (5/5) and + `${^LAST_SUCCESSFUL_PATTERN}` dynamic scope and reuse (25/25) on system + Perl, JVM, and interpreter; removed both obsolete `pat.t.patch` wrappers + and resynchronized the original assertions. + - [x] Preserved callback-bearing compiled regexes through one- and multi-item + array interpolation, including Perl's deferred dot-overload composition + with surrounding dynamic callbacks. The focused oracle passes 28/28 on + system Perl, JVM, and interpreter. + - [x] Removed the final `pat.t.patch` hunk, deleted the patch and its importer + configuration, and resynchronized the unmodified Perl 5.44 `pat.t`. - [ ] Phase 3: Unicode and pattern syntax completion (in progress) - [x] Added Perl escape syntax, Unicode-property resolution, scoped ASCII folds, possessive intervals, and bounded lookbehind support to Joni. + - [x] Converted public regex `pos` values between Perl logical-character + offsets and Java matcher offsets for scalar `/g`, `\G`, fast scanners, and + substitution callbacks. The 11-assertion supplementary-character oracle + passes on system Perl, JVM, and interpreter. + - [x] Restricted user-defined property dispatch to Perl's exact `Is`/`In` + naming convention and made unknown-property diagnostics fatal even in + compatibility warning mode. The focused oracle passes 39/39 on system + Perl, JVM, and interpreter; `regexp_unicode_prop.t` gains 15 assertions. - [ ] Complete the remaining Unicode aliases and diagnostic parity inventory. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) @@ -291,19 +366,17 @@ the remaining five inspect Perl's optimizer/debug transcript. ### Next Steps -1. Capture forced-Java and forced-Joni results for the full 80-file direct regex - corpus, compare both against PR 958, and classify any newly exposed gaps. -2. Run the applicable `pat_advanced.t` control-verb and condition sections, - then close Phase 2 if their direct and interpreter results agree. -3. Map each remaining `pat.t.patch` hunk to its semantic blocker. As each blocker - closes, remove its hunk and rerun the targeted importer so validation uses - the original Perl 5.44 assertions. -4. Capture the clean-branch JVM and interpreter 80-file baselines and compare +1. Remediate the seven reduced forced-Joni failure causes, starting with the + catastrophic `regexp*` cluster and the now-profiled `pat_psycho*` matcher + reconstruction path; keep per-child hard limits throughout. +2. Integrate the stacked Joni offset-map, regex-set property, and CEC timeout + slices, then rerun the forced-Joni corpus from the new combined head. +3. Capture the clean-branch JVM and interpreter 80-file baselines and compare both to PR 958 with the regression exit gate. -5. Inventory the remaining uncommon Unicode aliases and invalid-property - diagnostics against Perl 5.44's bundled tables, then move the next - matcher-semantic preprocessor slice into Joni. -6. Keep the warning-free whole-unit-suite gate green with Joni as the default; +4. Integrate the exact invalid-property dispatch/diagnostic slice, then finish + PerlOnJava4's built-in Unicode-alias slice and the six residual + user-property-definition diagnostic assertions against Perl 5.44 behavior. +5. Keep the warning-free whole-unit-suite gate green with Joni as the default; use explicit Java mode only to classify corpus regressions before Phase 5 removes the legacy backend and selector. @@ -314,11 +387,15 @@ the remaining five inspect Perl's optimizer/debug transcript. - Resource-sensitive baselines must wait for unrelated Java builds to finish. - The interpreter does not reliably expose the lexical package through `InterpreterState.currentPackage` while a regex executes. Localized - `$REGMARK`/`$REGERROR` slots are therefore discovered by their active - `GlobalRuntimeScalar` identities; a future regex call-site metadata field can - make nested simultaneous localizations exact without scanning active globals. + `$REGMARK`/`$REGERROR` slots are therefore enumerated from active dynamic + `GlobalRuntimeScalar` bindings rather than inferred from the current package + or scanned from dormant globals. - Shared parser or `eval` failures are fixed in focused slices when they block a regex semantic test, rather than being approximated inside the matcher. +- Starting a forced-Joni global match exactly on a supplementary character + also requires PR #1008's high-surrogate offset-map correction. The public + `pos` conversion is independently complete; add that exact-start assertion + when #1008 integrates. ## Related Documents and Skills diff --git a/dev/import-perl5/config.yaml b/dev/import-perl5/config.yaml index 6e6cabde7b..beea16f400 100644 --- a/dev/import-perl5/config.yaml +++ b/dev/import-perl5/config.yaml @@ -111,7 +111,6 @@ imports: - source: perl5/t/re/pat.t target: perl5_t/t/re/pat.t - patch: pat.t.patch - source: perl5/t/porting/manifest.t target: perl5_t/t/porting/manifest.t diff --git a/dev/import-perl5/patches/pat.t.patch b/dev/import-perl5/patches/pat.t.patch deleted file mode 100644 index 9baeff974e..0000000000 --- a/dev/import-perl5/patches/pat.t.patch +++ /dev/null @@ -1,109 +0,0 @@ ---- perl5/t/re/pat.t 2025-10-13 09:35:44 -+++ t/re/pat.t 2025-10-22 09:57:03 -@@ -714,7 +714,7 @@ - $_ = "aced"; - #12 3 4 5 - /((a?(*ACCEPT)())())()/ -- or die "Failed to match"; -+ or warn "Failed to match"; - is($1,"a",$message . "buffer 1 is defined with expected value"); - is($2,"a",$message . "buffer 2 is defined with expected value"); - ok(!defined($3),$message . "buffer 3 is not defined"); -@@ -723,7 +723,7 @@ - ok(!defined($6),$message . "buffer 6 is not defined"); - $message= 'NO ACCEPT and CLOSE - '; - /((a?())())()/ -- or die "Failed to match"; -+ or warn "Failed to match"; - is($1,"a",$message . "buffer 1 is defined with expected value"); - is($2,"a",$message . "buffer 2 is defined with expected value"); - is($3,"", $message . "buffer 3 is defined with expected value"); -@@ -733,7 +733,7 @@ - #12 3 4 5 - $message = 'ACCEPT and CLOSE - '; - /((a?(*ACCEPT)(c))(e))(d)/ -- or die "Failed to match"; -+ or warn "Failed to match"; - is($1,"a",$message . "buffer 1 is defined with expected value"); - is($2,"a",$message . "buffer 2 is defined with expected value"); - ok(!defined($3),$message . "buffer 3 is not defined"); -@@ -742,7 +742,7 @@ - ok(!defined($6),$message . "buffer 6 is not defined"); - $message= 'NO ACCEPT and CLOSE - '; - /((a?(c))(e))(d)/ -- or die "Failed to match"; -+ or warn "Failed to match"; - is($1,"ace", $message . "buffer 1 is defined with expected value"); - is($2,"ac", $message . "buffer 2 is defined with expected value"); - is($3,"c", $message . "buffer 3 is defined with expected value"); -@@ -897,7 +897,7 @@ - } - - -- { -+ eval { - my $message = 'pos inside (?{ })'; - my $str = 'abcde'; - our ($foo, $bar); -@@ -959,7 +959,7 @@ - "'abcd' 'e|' 'abcde' " . - "'abcde|' 'ab' 'cde' " . - "'abcde|' 'abc' 'de'", $message); -- } -+ } or fail("pos inside (?{ }) - not implemented"); - - { - my $message = '\G anchor checks'; -@@ -1183,11 +1183,11 @@ - sub new {bless []} - - my $message = "Ref stringification"; -- ::ok(do { \my $v} =~ /^SCALAR/, "Scalar ref stringification") or diag($message); -- ::ok(do {\\my $v} =~ /^REF/, "Ref ref stringification") or diag($message); -- ::ok([] =~ /^ARRAY/, "Array ref stringification") or diag($message); -- ::ok({} =~ /^HASH/, "Hash ref stringification") or diag($message); -- ::ok('S' -> new =~ /^Object S/, "Object stringification") or diag($message); -+ ::ok(do { \my $v} =~ /^SCALAR/, "Scalar ref stringification") or ::diag($message); -+ ::ok(do {\\my $v} =~ /^REF/, "Ref ref stringification") or ::diag($message); -+ ::ok([] =~ /^ARRAY/, "Array ref stringification") or ::diag($message); -+ ::ok({} =~ /^HASH/, "Hash ref stringification") or ::diag($message); -+ ::ok('S' -> new =~ /^Object S/, "Object stringification") or ::diag($message); - } - - { -@@ -2461,7 +2461,7 @@ - "GH Issue #18865 'XaaXbbXb' - test optimization"); - } - } -- { -+ eval { - # Test that ${^LAST_SUCCESSFUL_PATTERN} works as expected. - # It should match like the empty pattern does, and it should be dynamic - # in the same was as $1 is dynamic. -@@ -2511,7 +2511,7 @@ - ok($str =~ s/bar//,"matched bar"); - ok($str =~ s/$copy/PQR/, 'replaced $copy with PQR'); - is($str, "PQR", 'final string should be PQR'); -- } -+ } or fail('${^LAST_SUCCESSFUL_PATTERN} - not implemented'); - - - # Various tests for regexes with code blocks interpolated from an -@@ -2521,7 +2521,7 @@ - # "eval"', or would assert fail, or crash, or produce unpredictable - # results. - -- { -+ eval { - local $" = '-'; # separator when interpolating arrays - - my $pat; -@@ -2625,7 +2625,7 @@ - unlike "", $pat, "code in array 16 not 1"; - unlike "XAB-B-=EC", $pat, "code in array 16 not 2"; - -- } -+ } or fail("(??{}) code blocks in arrays - not implemented"); - - { - # github #21661 diff --git a/src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java b/src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java index daca9cc9c2..c36c33db27 100644 --- a/src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java +++ b/src/main/java/org/perlonjava/backend/bytecode/BytecodeCompiler.java @@ -3616,6 +3616,19 @@ void compileVariableDeclaration(OperatorNode node, String op) { boolean isDeclaredReference = node.annotations != null && Boolean.TRUE.equals(node.annotations.get("isDeclaredReference")); + // A later `our $x` under a different package establishes + // a new lexical package alias even when an outer/sibling + // declaration reused the same bare name. Refresh the + // symbol entry as well as its register value; otherwise + // later reads keep qualifying $x with the old package. + if (hasVariable(varName) && isOurVariable(varName)) { + SymbolTable.SymbolEntry entry = symbolTable.getSymbolEntry(varName); + if (entry != null + && !getCurrentPackage().equals(entry.perlPackage())) { + addVariable(varName, "our"); + } + } + // Check if already declared in current scope if (hasVariable(varName)) { int reg = getVariableRegister(varName); diff --git a/src/main/java/org/perlonjava/runtime/operators/PerlUtfString.java b/src/main/java/org/perlonjava/runtime/operators/PerlUtfString.java index 2671daa50e..47839cd661 100644 --- a/src/main/java/org/perlonjava/runtime/operators/PerlUtfString.java +++ b/src/main/java/org/perlonjava/runtime/operators/PerlUtfString.java @@ -161,6 +161,28 @@ public static int offsetByPerlCodePoints(String s, int startJava, int perlOffset return scanOffsetByPerlCodePoints(s, startJava, perlOffset); } + /** Convert a Java UTF-16 boundary to its Perl logical-character offset. */ + public static int perlOffsetForJavaIndex(String s, int javaIndex) { + int bounded = Math.max(0, Math.min(javaIndex, s.length())); + PerlIndexMap indexMap = cachedIndexMap(s); + if (indexMap != null) { + int index = Arrays.binarySearch(indexMap.javaBoundaries, bounded); + if (index >= 0) return index; + // Matcher offsets should be logical boundaries. If an adapter + // supplies an interior UTF-16 index, use the preceding boundary. + return Math.max(0, -index - 2); + } + int logical = 0; + int current = 0; + while (current < bounded) { + int next = readOnePerlLogical(s, current).nextJavaIndex(); + if (next > bounded) break; + current = next; + logical++; + } + return logical; + } + private static int scanOffsetByPerlCodePoints(String s, int startJava, int perlOffset) { int j = startJava; for (int k = 0; k < perlOffset && j < s.length(); k++) { diff --git a/src/main/java/org/perlonjava/runtime/operators/StringOperators.java b/src/main/java/org/perlonjava/runtime/operators/StringOperators.java index 7d69ce4024..d926b33b31 100644 --- a/src/main/java/org/perlonjava/runtime/operators/StringOperators.java +++ b/src/main/java/org/perlonjava/runtime/operators/StringOperators.java @@ -902,7 +902,17 @@ private static RuntimeScalar joinInternal(RuntimeScalar runtimeScalar, RuntimeBa WarnDie.warnWithCategory(new RuntimeScalar("Use of uninitialized value in join or string"), RuntimeScalarCache.scalarEmptyString, "uninitialized"); } - RuntimeScalar resolved = stringifyForStringContext(resolveTiedStringOperand(scalar)); + RuntimeScalar resolved = resolveTiedStringOperand(scalar); + // A one-element interpolated array must retain a compiled regex's + // callback template. Stringifying it here turns trusted (??{...}) + // callbacks back into runtime source and incorrectly requires + // lexical `use re 'eval'`. + if (isStringInterpolation && list instanceof RuntimeArray + && RuntimeRegexTemplate.hasExecutableValue(resolved)) { + return recordJoinTaint(RuntimeRegexTemplate.buildJoined( + runtimeScalar, java.util.List.of(resolved))); + } + resolved = stringifyForStringContext(resolved); RuntimeScalar res = new RuntimeScalar(resolved.toString()); if (resolved.type != RuntimeScalarType.STRING) { res.type = BYTE_STRING; @@ -946,10 +956,15 @@ private static RuntimeScalar joinInternal(RuntimeScalar runtimeScalar, RuntimeBa RuntimeScalarCache.scalarEmptyString, "uninitialized"); } - RuntimeScalar resolved = stringifyForStringContext(resolveTiedStringOperand(scalar)); + RuntimeScalar materialized = resolveTiedStringOperand(scalar); + RuntimeScalar resolved = stringifyForStringContext(materialized); if (isStringInterpolation) { - resolvedElements.add(resolved); - hasExecutableValue |= RuntimeRegexTemplate.hasExecutableValue(resolved); + // If any element carries executable regex callbacks, buildJoined + // must still see blessed elements so the outer regex-template + // concatenation can dispatch their `.` overload. Ordinary joins + // continue to use the stringified value accumulated below. + resolvedElements.add(materialized); + hasExecutableValue |= RuntimeRegexTemplate.hasExecutableValue(materialized); } if (resolved.type == RuntimeScalarType.STRING) { hasUtf8 = true; diff --git a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java index c22953bb8d..6d96310ece 100644 --- a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java +++ b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java @@ -72,8 +72,13 @@ final class JoniRegexPattern { } RegexMatcher matcher(String input, List callbacks) { + return matcher(input, callbacks, new RuntimeScalar(input)); + } + + RegexMatcher matcher(String input, List callbacks, + RuntimeScalar subject) { return new JoniRegexMatcher(regex, sourcePattern, namedGroups, flags, - hasControlVerbState, input, callbacks); + hasControlVerbState, input, callbacks, subject); } String patternDescription() { @@ -115,7 +120,7 @@ private static UserPropertyTranslation translateUserDefinedProperties( String property = pattern.substring(i + 3, end).trim(); String unnegated = property.startsWith("^") ? property.substring(1).trim() : property; - boolean userDefined = unnegated.matches("^(.*::)?([Ii][sSNn]).+"); + boolean userDefined = UnicodeResolver.isUserDefinedPropertyName(unnegated); boolean scriptExtensions = unnegated.matches( "(?i)^(?:scx|script[_ ]?extensions)\\s*=.*"); boolean frontendProperty = unnegated.matches( @@ -583,11 +588,12 @@ private static final class JoniRegexMatcher implements RegexMatcher { private int committedLastClosedCapture = -1; private final boolean hasControlVerbState; private final List callbacks; + private final RuntimeScalar subject; private PerlCalloutHandler calloutHandler; JoniRegexMatcher(Regex regex, String sourcePattern, Map namedGroups, RegexFlags flags, boolean hasControlVerbState, String input, - List callbacks) { + List callbacks, RuntimeScalar subject) { this.regex = regex; this.sourcePattern = sourcePattern; this.namedGroups = namedGroups; @@ -595,6 +601,7 @@ private static final class JoniRegexMatcher implements RegexMatcher { this.hasControlVerbState = hasControlVerbState; this.input = input; this.callbacks = callbacks; + this.subject = subject; this.bytes = input.getBytes(StandardCharsets.UTF_8); this.charToByte = buildCharToByte(input); this.byteToChar = buildByteToChar(input, bytes.length, charToByte); @@ -620,7 +627,8 @@ private boolean find(int option, boolean anchored) { } matcher = regex.matcher(bytes); if (!callbacks.isEmpty()) { - calloutHandler = new PerlCalloutHandler(input, byteToChar, callbacks, flags); + calloutHandler = new PerlCalloutHandler( + input, byteToChar, callbacks, flags, hasControlVerbState, subject); matcher.setCalloutHandler(calloutHandler); } int result; @@ -633,7 +641,7 @@ private boolean find(int option, boolean anchored) { throw failure; } matched = result >= 0; - if (hasControlVerbState) { + if (hasControlVerbState || matcher.hasEncounteredControlVerb()) { RuntimeRegex.updateControlVerbVariables( matcher.getControlMark(), matcher.getControlError()); } @@ -805,6 +813,8 @@ static CaptureSnapshot of(MatchView match) { private final int[] byteToChar; private final List callbacks; private final RegexFlags outerFlags; + private final boolean publishesControlVerbState; + private final RuntimeScalar subject; private final int initialLocalLevel; private final RegexState initialRegexState; private final PerlCalloutHandler parent; @@ -817,17 +827,22 @@ static CaptureSnapshot of(MatchView match) { private String failedNestedLastParenMatch; PerlCalloutHandler(String input, int[] byteToChar, List callbacks, - RegexFlags outerFlags) { - this(input, byteToChar, callbacks, outerFlags, null); + RegexFlags outerFlags, boolean publishesControlVerbState, + RuntimeScalar subject) { + this(input, byteToChar, callbacks, outerFlags, publishesControlVerbState, + subject, null); } private PerlCalloutHandler( String input, int[] byteToChar, List callbacks, - RegexFlags outerFlags, PerlCalloutHandler parent) { + RegexFlags outerFlags, boolean publishesControlVerbState, + RuntimeScalar subject, PerlCalloutHandler parent) { this.input = input; this.byteToChar = byteToChar; this.callbacks = callbacks; this.outerFlags = outerFlags; + this.publishesControlVerbState = publishesControlVerbState; + this.subject = subject; this.parent = parent; this.nestedDepth = parent == null ? 0 : parent.nestedDepth + 1; this.initialLocalLevel = DynamicVariableManager.getLocalLevel(); @@ -897,6 +912,8 @@ public DynamicPatternResult executeDynamic(int id, MatchView match) { value.value instanceof RuntimeRegex runtimeRegex && runtimeRegex.getRegexFlags() != null ? runtimeRegex.getRegexFlags() : outerFlags, + nestedPattern.hasControlVerbState, + subject, this); if (nestedHandler != null) executedNestedCallbackPattern = true; return new DynamicPatternResult(nestedPattern.engineRegex(), nestedHandler, @@ -939,6 +956,18 @@ private Evaluation evaluate(RuntimeRegexCallback callback, MatchView match) { var callbackPackages = PerlRuntime.current().executionState() .activeRegexCallbackPackages; callbackPackages.push(callback.lexicalPackage == null ? "main" : callback.lexicalPackage); + RuntimeScalar previousTopic = GlobalVariable.getGlobalVariable("main::_"); + boolean previousTopicWasTemporary = + GlobalVariable.isTemporaryGlobalAlias("main::_"); + RuntimeScalar callbackPosition = RuntimePosLvalue.pos(subject); + int previousPositionType = callbackPosition.type; + Object previousPositionValue = callbackPosition.value; + GlobalVariable.aliasTemporaryGlobalVariable("main::_", subject); + // Exposing the provisional offset is not an assignment to pos(). + // Preserve zero-length /g bookkeeping unless callback code itself + // explicitly changes pos(). + callbackPosition.type = RuntimeScalarType.INTEGER; + callbackPosition.value = charOffset(match.currentBytePosition()); try { DynamicVariableManager.CapturedFrame frame = @@ -967,6 +996,10 @@ private Evaluation evaluate(RuntimeRegexCallback callback, MatchView match) { previousDynamicView = priorDynamicView; throw failure; } finally { + callbackPosition.type = previousPositionType; + callbackPosition.value = previousPositionValue; + GlobalVariable.restoreTemporaryGlobalVariable( + "main::_", previousTopic, previousTopicWasTemporary); if (!callbackLocations.isEmpty()) callbackLocations.pop(); if (!callbackPackages.isEmpty()) callbackPackages.pop(); if (callback.code.isQuotedRegexCallback) { @@ -1091,9 +1124,15 @@ private void publishProvisional(MatchView match) { state.lastParenMatchOverride = null; int count = match.captureCount(); state.globalMatchString = input; - state.lastMatchedString = input; - state.lastMatchStart = charOffset(match.captureBegin(0)); - state.lastMatchEnd = charOffset(match.captureEnd(0)); + int provisionalStart = charOffset(match.captureBegin(0)); + int provisionalEnd = charOffset(match.captureEnd(0)); + provisionalEnd = provisionalEnd >= 0 + ? provisionalEnd : charOffset(match.currentBytePosition()); + state.lastMatchStart = provisionalStart; + state.lastMatchEnd = provisionalEnd; + state.lastMatchedString = provisionalStart >= 0 + && provisionalEnd >= provisionalStart + ? input.substring(provisionalStart, provisionalEnd) : null; state.lastCaptureGroups = new String[count]; state.manualCaptureStarts = new int[count]; state.manualCaptureEnds = new int[count]; @@ -1113,7 +1152,9 @@ private void publishProvisional(MatchView match) { int lastClosed = match.lastClosedCapture(); state.lastClosedCapture = lastClosed > 0 && lastClosed <= count ? state.lastCaptureGroups[lastClosed - 1] : null; - RuntimeRegex.updateControlVerbVariables(match.controlMark(), null); + if (publishesControlVerbState || match.controlMark() != null) { + RuntimeRegex.updateControlVerbVariables(match.controlMark(), null); + } } private int charOffset(int byteOffset) { diff --git a/src/main/java/org/perlonjava/runtime/regex/RegexPreprocessorHelper.java b/src/main/java/org/perlonjava/runtime/regex/RegexPreprocessorHelper.java index a87a69510e..e289d53653 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RegexPreprocessorHelper.java +++ b/src/main/java/org/perlonjava/runtime/regex/RegexPreprocessorHelper.java @@ -360,7 +360,8 @@ static int handleEscapeSequences(String s, StringBuilder sb, int c, int offset, // But if the error already contains "in expansion of", it is a real user-property definition error // that should be reported (not deferred). String msg = e.getMessage(); - if (property.matches("^(.*::)?([Ii][sSNn]).+") && (msg == null || !msg.contains("in expansion of"))) { + if (UnicodeResolver.isUserDefinedPropertyName(property) + && (msg == null || !msg.contains("in expansion of"))) { RegexPreprocessor.markDeferredUnicodePropertyEncountered(); sb.setLength(sb.length() - 1); // Remove the backslash // Placeholder: match any single character, including newline diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java index 2ce6f86846..77cf10b877 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java @@ -65,7 +65,9 @@ public static RuntimeScalar stabilizeLiteralTarget(RuntimeScalar literal, int ca private static final int MULTILINE = Pattern.MULTILINE; private static final int DOTALL = Pattern.DOTALL; private static final Pattern USER_DEFINED_PROPERTY_PATTERN = - Pattern.compile("\\\\([pP])\\{((?:[A-Za-z_][A-Za-z0-9_]*::)*(?:[Ii][sS]|[Ii][nN])[A-Za-z0-9_]*)}"); + Pattern.compile("\\\\([pP])\\{((?:[A-Za-z_][A-Za-z0-9_]*::)*(?:Is|In)[A-Za-z_][A-Za-z0-9_]*)}"); + private static final Pattern MALFORMED_USER_DEFINED_PROPERTY_PATTERN = + Pattern.compile("(?:^|::)(?:Is|In)::"); private static final Pattern UNICODE_PROPERTY_PATTERN = Pattern.compile("\\\\[pP]\\{"); // Maximum size for each runtime's regex cache. @@ -157,8 +159,7 @@ static void updateControlVerbVariables(String mark, String error) { // facade synchronized with every lexical package statement, so use the // localized scalar identities rather than guessing one package name. for (Map.Entry entry - : new ArrayList<>(GlobalVariable.globalVariables.entrySet())) { - if (!(entry.getValue() instanceof GlobalRuntimeScalar)) continue; + : DynamicVariableManager.activeLocalizedGlobalScalars().entrySet()) { if (entry.getKey().endsWith("::REGMARK")) { entry.getValue().set(markValue); } else if (entry.getKey().endsWith("::REGERROR")) { @@ -302,7 +303,7 @@ public Pattern selectPattern(RuntimeScalar string) { */ public RegexMatcher matcher(RuntimeScalar string, String input) { if (recursivePattern != null) { - return selectRecursivePattern(string).matcher(input, executableCallbacks); + return selectRecursivePattern(string).matcher(input, executableCallbacks, string); } Pattern selected = selectPattern(string, input); return new JavaRegexMatcher( @@ -507,7 +508,7 @@ private static RuntimeRegex compile(String patternString, String modifiers, int public static boolean requiresRuntimeUnicodePropertyResolution(String patternString) { if (patternString == null) return false; return java.util.regex.Pattern.compile( - "\\\\[pP]\\{\\^?\\s*(?:[^}]*::)?[Ii][sSnN][^}]+}") + "\\\\[pP]\\{\\^?\\s*(?:[A-Za-z_][A-Za-z0-9_]*::)*(?:Is|In)[A-Za-z_][A-Za-z0-9_]*\\s*}") .matcher(patternString).find(); } @@ -734,6 +735,15 @@ private static synchronized RuntimeRegex compileSynchronized( || "invalid backref number".equals(e.getMessage())) { throw new PerlCompilerException("Reference to nonexistent group"); } + String invalidProperty = invalidUnicodePropertyName(e.getMessage()); + if (invalidProperty != null) { + if (MALFORMED_USER_DEFINED_PROPERTY_PATTERN.matcher(invalidProperty).find()) { + throw new PerlCompilerException( + "Illegal user-defined property name \"" + invalidProperty + "\""); + } + throw new PerlCompilerException( + "Can't find Unicode property definition \"" + invalidProperty + "\""); + } // PerlJavaUnimplementedException extends PerlCompilerException, so check // the more specific type first. Real syntax errors (PerlCompilerException // but NOT PerlJavaUnimplementedException) are always fatal. @@ -792,6 +802,16 @@ private static synchronized RuntimeRegex compileSynchronized( return regex; } + private static String invalidUnicodePropertyName(String message) { + if (message == null) return null; + String prefix = "invalid character property name <"; + int start = message.indexOf(prefix); + if (start < 0) return null; + start += prefix.length(); + int end = message.indexOf('>', start); + return end < 0 ? null : message.substring(start, end); + } + private static int debugMode(String modifiers) { if (modifiers == null) return 0; if (modifiers.indexOf(INTERNAL_DEBUGCOLOR_MARKER) >= 0) return 2; @@ -1600,7 +1620,8 @@ public static RuntimeScalar getReplacementRegex(RuntimeScalar patternString, Run // Only recompile if flags actually changed (this is needed for /x preprocessing) if (flagsChanged && !resolved.fromCompiledRegex()) { RuntimeRegex recompiledRegex = compile(resolvedRegex.patternString, - newFlags.toFlagString(), regex.lexicalDebugMode); + newFlags.toFlagString(), regex.lexicalDebugMode, + resolvedRegex.executableCallbacks.size()); regex.pattern = recompiledRegex.pattern; regex.patternUnicode = recompiledRegex.patternUnicode; regex.recursivePattern = recompiledRegex.recursivePattern; @@ -1886,7 +1907,7 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc RegexMatcher matcher; if (regex.recursivePattern != null) { matcher = regex.selectRecursivePattern(string) - .matcher(inputStr, regex.executableCallbacks); + .matcher(inputStr, regex.executableCallbacks, string); } else { Pattern pattern = regex.selectPattern(string, inputStr); @@ -1914,7 +1935,9 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc // Use RuntimePosLvalue to get the current position posScalar = RuntimePosLvalue.pos(string); isPosDefined = posScalar.getDefinedBoolean(); - startPos = isPosDefined ? posScalar.getInt() : 0; + startPos = isPosDefined + ? RuntimePosLvalue.toMatcherOffset(string, inputStr, posScalar.getInt()) + : 0; RuntimeBase fastXmlElementResult = tryFastXmlElementScan(regex, string, inputStr, startPos, posScalar, ctx); if (fastXmlElementResult != null) { @@ -1934,7 +1957,8 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc // adds (?=[\s\S]) to prevent matching at end of string. if (regex.regexFlags.isGlobalMatch() && ctx == RuntimeContextType.SCALAR) { String patternKey = regex.patternString; - if (RuntimePosLvalue.hadZeroLengthMatchAt(string, startPos, patternKey)) { + if (RuntimePosLvalue.hadZeroLengthMatchAt( + string, posScalar.getInt(), patternKey)) { // First, try the notempty variant at the SAME position (Perl behavior) boolean notemptySucceeded = false; if (regex.notemptyPattern != null) { @@ -1967,7 +1991,7 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc if (!notemptySucceeded) { // Notempty variant didn't find a match; fall back to bumpalong - startPos++; + startPos = bumpGlobalMatchPosition(inputStr, startPos); if (startPos > inputStr.length()) { // Past end of string, fail if (!regex.regexFlags.keepCurrentPosition()) { @@ -1975,7 +1999,8 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc } return RuntimeScalarCache.scalarFalse; } - posScalar.set(startPos); + posScalar.set(RuntimePosLvalue.fromMatcherOffset( + string, inputStr, startPos)); RuntimePosLvalue.recordNonZeroLengthMatch(string); isPosDefined = true; } @@ -2096,7 +2121,7 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc break; } // In middle of string, advance by 1 to avoid infinite loop - matchEnd = matchStart + 1; + matchEnd = bumpGlobalMatchPosition(inputStr, matchStart); forcedAdvance = true; } @@ -2106,10 +2131,13 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc // Set pos to the end of the current match to prepare for the next search // (only for global matches - posScalar is null for non-global) if (posScalar != null) { - posScalar.set(matchEnd); + int perlMatchEnd = RuntimePosLvalue.fromMatcherOffset( + string, inputStr, matchEnd); + posScalar.set(perlMatchEnd); // Record zero-length match for cross-call tracking if (matchEnd == matchStart) { - RuntimePosLvalue.recordZeroLengthMatch(string, matchEnd, regex.patternString); + RuntimePosLvalue.recordZeroLengthMatch( + string, perlMatchEnd, regex.patternString); } else { RuntimePosLvalue.recordNonZeroLengthMatch(string); } @@ -2118,7 +2146,8 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc } else { startPos = matchEnd; if (posScalar != null) { - posScalar.set(startPos); + posScalar.set(RuntimePosLvalue.fromMatcherOffset( + string, inputStr, startPos)); } // Only redirect the matcher when we forcibly advanced past // a zero-length match. In every other case Java's find() @@ -2305,7 +2334,8 @@ private static RuntimeBase tryFastXmpMetaElementScan(RuntimeRegex regex, regex.matched = true; if (regex.regexFlags.isGlobalMatch() && posScalar != null) { - posScalar.set(state().lastMatchEnd); + posScalar.set(RuntimePosLvalue.fromMatcherOffset( + string, inputStr, state().lastMatchEnd)); RuntimePosLvalue.recordNonZeroLengthMatch(string); } @@ -2377,7 +2407,8 @@ private static RuntimeBase tryFastXmlElementScan(RuntimeRegex regex, regex.matched = true; if (regex.regexFlags.isGlobalMatch() && posScalar != null) { - posScalar.set(state().lastMatchEnd); + posScalar.set(RuntimePosLvalue.fromMatcherOffset( + string, inputStr, state().lastMatchEnd)); RuntimePosLvalue.recordNonZeroLengthMatch(string); } @@ -2728,7 +2759,7 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar RegexMatcher matcher; if (regex.recursivePattern != null) { matcher = regex.selectRecursivePattern(inputValue) - .matcher(inputStr, regex.executableCallbacks); + .matcher(inputStr, regex.executableCallbacks, inputValue); } else { pattern = regex.selectPattern(inputValue, inputStr); if (inputStr.isEmpty() && (pattern.flags() & Pattern.MULTILINE) != 0) { @@ -2751,7 +2782,8 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar if (regex.useGAssertion) { RuntimeScalar posScalar = RuntimePosLvalue.pos(string); if (posScalar.getDefinedBoolean()) { - int startPos = posScalar.getInt(); + int startPos = RuntimePosLvalue.toMatcherOffset( + string, inputStr, posScalar.getInt()); if (startPos >= 0 && startPos <= inputStr.length()) { searchStart = startPos; } @@ -2801,7 +2833,8 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar // non-destructive /r substitution leaves the target's // original pos untouched. if (destructiveReplacement) { - RuntimePosLvalue.pos(string).set(matcher.start()); + RuntimePosLvalue.pos(string).set(RuntimePosLvalue.fromMatcherOffset( + string, inputStr, matcher.start())); } // Use callerArgs (the enclosing subroutine's @_) so $_[0] etc. work RuntimeArray args = (callerArgs != null) ? callerArgs : new RuntimeArray(); @@ -2860,7 +2893,7 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar ? new JavaRegexMatcher(nonEmptySubstitutionPattern.matcher(matchInput), regex.branchResetCaptureMap) : regex.selectRecursivePattern(inputValue) - .matcher(inputStr, regex.executableCallbacks); + .matcher(inputStr, regex.executableCallbacks, inputValue); // The synthetic (?<=[\s\S]) suffix relies on opaque bounds // so a zero-length match at the region start is rejected. setSubstitutionRegion(retryMatcher, zeroLengthOffset, inputStr.length(), false); @@ -2875,7 +2908,8 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar String retryReplacementStr; if (replacementIsCode) { if (destructiveReplacement) { - RuntimePosLvalue.pos(string).set(retryMatcher.start()); + RuntimePosLvalue.pos(string).set(RuntimePosLvalue.fromMatcherOffset( + string, inputStr, retryMatcher.start())); } RuntimeArray args = (callerArgs != null) ? callerArgs : new RuntimeArray(); RuntimeList result = RuntimeCode.apply(replacement, args, RuntimeContextType.SCALAR); diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegexTemplate.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegexTemplate.java index e6ac1b68e2..45a81e94f5 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegexTemplate.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegexTemplate.java @@ -1,5 +1,6 @@ package org.perlonjava.runtime.regex; +import org.perlonjava.runtime.operators.StringOperators; import org.perlonjava.runtime.runtimetypes.RuntimeBase; import org.perlonjava.runtime.runtimetypes.RuntimeList; import org.perlonjava.runtime.runtimetypes.RuntimeScalar; @@ -17,6 +18,20 @@ public final class RuntimeRegexTemplate { private final String pattern; private final List callbacks; + /** + * Deferred array interpolation. Keeping the joined operands separate until + * the enclosing regex template is assembled lets a blessed element's dot + * overload observe the complete left-hand pattern, as Perl does. + */ + private record JoinedParts(List parts) { + @Override + public String toString() { + StringBuilder joined = new StringBuilder(); + for (RuntimeScalar part : parts) joined.append(part); + return joined.toString(); + } + } + record MaskedCallouts(String pattern, String syntheticPrefix, List placeholders, List markers) { String restore(String compiledPattern) { @@ -37,6 +52,7 @@ private RuntimeRegexTemplate(String pattern, List callback } public static RuntimeScalar build(RuntimeList parts) { + parts = flattenJoinedParts(parts); if (parts.elements.size() == 1) { RuntimeScalar only = parts.elements.getFirst().scalar(); // A lone interpolation must retain its runtime type so qr @@ -44,6 +60,24 @@ public static RuntimeScalar build(RuntimeList parts) { // Only a parser-created callback needs a new template skeleton. if (!(only.value instanceof RuntimeRegexCallback)) return only; } + boolean hasBlessedPart = false; + for (RuntimeBase part : parts.elements) { + if (RuntimeScalarType.blessedId(part.scalar()) != 0) { + hasBlessedPart = true; + break; + } + } + if (hasBlessedPart) { + RuntimeScalar concatenated = new RuntimeScalar(""); + for (RuntimeBase part : parts.elements) { + RuntimeScalar scalar = part.scalar(); + if (scalar.value instanceof RuntimeRegexCallback callback) { + scalar = new RuntimeScalar(callback.source == null ? "" : callback.source); + } + concatenated = StringOperators.stringConcat(concatenated, scalar); + } + return concatenated; + } StringBuilder pattern = new StringBuilder(); List callbacks = new ArrayList<>(); boolean tainted = false; @@ -84,20 +118,39 @@ public static RuntimeScalar build(RuntimeList parts) { /** Preserve callback-bearing qr values while an array is joined for interpolation. */ public static RuntimeScalar buildJoined(RuntimeScalar separator, List elements) { - RuntimeList parts = new RuntimeList(); + List parts = new ArrayList<>(); + boolean tainted = separator.isTainted(); for (int i = 0; i < elements.size(); i++) { if (i > 0) parts.add(separator); - parts.add(elements.get(i)); + RuntimeScalar element = elements.get(i); + parts.add(element); + tainted |= element.isTainted(); } - return build(parts); + RuntimeScalar result = new RuntimeScalar(new JoinedParts(List.copyOf(parts))); + result.tainted = tainted; + return result; } public static boolean hasExecutableValue(RuntimeScalar scalar) { - return scalar != null && (scalar.value instanceof RuntimeRegexTemplate + return scalar != null && (scalar.value instanceof JoinedParts + || scalar.value instanceof RuntimeRegexTemplate || scalar.value instanceof RuntimeRegex regex && !regex.executableCallbacks.isEmpty()); } + private static RuntimeList flattenJoinedParts(RuntimeList input) { + RuntimeList flattened = new RuntimeList(); + for (RuntimeBase part : input.elements) { + RuntimeScalar scalar = part.scalar(); + if (scalar.value instanceof JoinedParts joined) { + for (RuntimeScalar joinedPart : joined.parts) flattened.add(joinedPart); + } else { + flattened.add(part); + } + } + return flattened; + } + private static void appendEmbeddedRegex(StringBuilder pattern, List callbacks, String embeddedPattern, diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index 566fe4c97f..1b92bdd941 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -9,8 +9,12 @@ import java.util.HashSet; import java.util.Map; import java.util.Set; +import java.util.regex.Pattern; public class UnicodeResolver { + private static final Pattern USER_DEFINED_PROPERTY_NAME = Pattern.compile( + "^(?:[A-Za-z_][A-Za-z0-9_]*::)*(?:Is|In)[A-Za-z_][A-Za-z0-9_]*$"); + private static final String[] PERL_UNICODE_AGE_VERSIONS = { "1.1", "2.0", "2.1", "3.0", "3.1", "3.2", "4.0", "4.1", "5.0", "5.1", "5.2", "6.0", "6.1", "6.2", "6.3", "7.0", @@ -610,7 +614,7 @@ static void preloadUserDefinedProperties(String pattern, boolean caseInsensitive String property = pattern.substring(slash + 3, end).trim(); if (property.startsWith("^")) property = property.substring(1).trim(); - if (property.matches("^(.*::)?([Ii][sSNn]).+")) { + if (isUserDefinedPropertyName(property)) { // Preloading is an optimization for callbacks that are already // available. Perl permits qr// to contain a forward reference // to a user property; RegexPreprocessor represents that with a @@ -656,11 +660,9 @@ private static String translateUnicodeProperty(String property, boolean negated, } } - // Check for user-defined properties (Is... or In...) - // Perl treats ANY property starting with Is/In (case-insensitive prefix) - // as potentially user-defined, regardless of the character after the prefix - // (e.g., Is_q, IsMyProp, InMyBlock all trigger user-defined lookup) - if (property.matches("^(.*::)?([Ii][sSNn]).+")) { + // User-defined properties require an exact Is/In prefix on the final + // identifier. Lowercase variants are ordinary unknown properties. + if (isUserDefinedPropertyName(property)) { String userProp = tryUserDefinedProperty( property, recursionSet, caseInsensitive); if (userProp != null) { @@ -875,6 +877,10 @@ private static String translateUnicodeProperty(String property, boolean negated, } } + static boolean isUserDefinedPropertyName(String property) { + return property != null && USER_DEFINED_PROPERTY_NAME.matcher(property).matches(); + } + private static String translatePerlAgeProperty(String property, boolean negated) { int equals = property.indexOf('='); if (equals <= 0 || equals == property.length() - 1) return null; diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/DynamicVariableManager.java b/src/main/java/org/perlonjava/runtime/runtimetypes/DynamicVariableManager.java index 8d64648cfa..d6eae49ca1 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/DynamicVariableManager.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/DynamicVariableManager.java @@ -3,7 +3,9 @@ import java.util.ArrayDeque; import java.util.ArrayList; import java.util.Deque; +import java.util.LinkedHashMap; import java.util.List; +import java.util.Map; import java.util.function.Supplier; /** @@ -55,6 +57,17 @@ public static boolean isGlobalScalarLocalized(String fullName) { return false; } + /** Return active package-scalar localizations by dynamic binding identity. */ + public static Map activeLocalizedGlobalScalars() { + Map localized = new LinkedHashMap<>(); + for (DynamicState state : variableStack()) { + if (!(state instanceof GlobalRuntimeScalar scalar)) continue; + RuntimeScalar value = scalar.activeLocalizedValue(); + if (value != null) localized.put(scalar.localizedName(), value); + } + return localized; + } + /** * Pushes a new dynamic variable onto the stack. This method saves the current state * of the variable before pushing it onto the stack, allowing it to be restored later. diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/GlobalRuntimeScalar.java b/src/main/java/org/perlonjava/runtime/runtimetypes/GlobalRuntimeScalar.java index 68d492edb0..19e75033fb 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/GlobalRuntimeScalar.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/GlobalRuntimeScalar.java @@ -26,6 +26,21 @@ boolean localizes(String variableName) { return fullName.equals(variableName); } + String localizedName() { + return fullName; + } + + RuntimeScalar activeLocalizedValue() { + Stack states = localizedStack(); + for (int i = states.size() - 1; i >= 0; i--) { + SavedGlobalState state = states.get(i); + if (state.fullName.equals(fullName)) { + return state.localizedVariable; + } + } + return null; + } + public static RuntimeScalar makeLocal(String fullName) { RuntimeScalar original = GlobalVariable.getGlobalVariable(fullName); if (original instanceof ScalarSpecialVariable sv && sv.variableId == ScalarSpecialVariable.Id.INPUT_LINE_NUMBER) { diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimePosLvalue.java b/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimePosLvalue.java index 2ca44664f3..d1ee422898 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimePosLvalue.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimePosLvalue.java @@ -1,5 +1,7 @@ package org.perlonjava.runtime.runtimetypes; +import org.perlonjava.runtime.operators.PerlUtfString; + import java.util.Map; /** @@ -46,6 +48,24 @@ public static RuntimeScalar pos(RuntimeScalar perlVariable) { return position; } + /** Convert a public Perl pos() value to the matcher's Java UTF-16 offset. */ + public static int toMatcherOffset(RuntimeScalar perlVariable, + String stringValue, int perlPosition) { + if (perlVariable.type == RuntimeScalarType.BYTE_STRING) { + return Math.max(0, Math.min(perlPosition, stringValue.length())); + } + return PerlUtfString.offsetByPerlCodePoints(stringValue, 0, perlPosition); + } + + /** Convert a matcher Java UTF-16 offset to the public Perl pos() value. */ + public static int fromMatcherOffset(RuntimeScalar perlVariable, + String stringValue, int matcherOffset) { + if (perlVariable.type == RuntimeScalarType.BYTE_STRING) { + return Math.max(0, Math.min(matcherOffset, stringValue.length())); + } + return PerlUtfString.perlOffsetForJavaIndex(stringValue, matcherOffset); + } + /** Copy pos() and zero-length /g bookkeeping between equivalent scalar views. */ public static void copyPositionState(RuntimeScalar source, RuntimeScalar target) { RuntimeScalar targetPosition = pos(target); diff --git a/src/test/resources/unit/our_package_redeclaration.t b/src/test/resources/unit/our_package_redeclaration.t new file mode 100644 index 0000000000..3ce8988dfe --- /dev/null +++ b/src/test/resources/unit/our_package_redeclaration.t @@ -0,0 +1,23 @@ +use strict; +use warnings; +use Test::More tests => 4; + +{ + package OurPackageFirst; + our $slot = 'first'; + + package OurPackageSecond; + our $slot = 'second'; + + ::is($slot, 'second', + 'redeclared our scalar reads from its new package'); + { + local $slot = 'localized second'; + ::is($slot, 'localized second', + 'local uses the redeclared our package'); + } + ::is($slot, 'second', + 'redeclared our scalar restores its package value'); + ::is($OurPackageFirst::slot, 'first', + 'the earlier package scalar remains independent'); +} diff --git a/src/test/resources/unit/regex/accept_nested_capture_close.t b/src/test/resources/unit/regex/accept_nested_capture_close.t new file mode 100644 index 0000000000..699fbc6ecb --- /dev/null +++ b/src/test/resources/unit/regex/accept_nested_capture_close.t @@ -0,0 +1,37 @@ +use strict; +use warnings; +use Test::More tests => 28; + +$_ = 'aced'; + +ok(/((a?(*ACCEPT)())())()/, 'nested ACCEPT matches'); +is($1, 'a', 'ACCEPT closes outer capture 1'); +is($2, 'a', 'ACCEPT closes outer capture 2'); +ok(!defined($3), 'ACCEPT leaves following capture 3 undefined'); +ok(!defined($4), 'ACCEPT leaves following capture 4 undefined'); +ok(!defined($5), 'ACCEPT leaves following capture 5 undefined'); +ok(!defined($6), 'ACCEPT leaves following capture 6 undefined'); + +ok(/((a?())())()/, 'equivalent pattern without ACCEPT matches'); +is($1, 'a', 'ordinary close captures group 1'); +is($2, 'a', 'ordinary close captures group 2'); +is($3, '', 'ordinary close captures group 3'); +is($4, '', 'ordinary close captures group 4'); +is($5, '', 'ordinary close captures group 5'); +ok(!defined($6), 'ordinary close leaves group 6 undefined'); + +ok(/((a?(*ACCEPT)(c))(e))(d)/, 'ACCEPT bypasses required suffixes'); +is($1, 'a', 'suffix-bypassing ACCEPT closes capture 1'); +is($2, 'a', 'suffix-bypassing ACCEPT closes capture 2'); +ok(!defined($3), 'bypassed capture 3 is undefined'); +ok(!defined($4), 'bypassed capture 4 is undefined'); +ok(!defined($5), 'bypassed capture 5 is undefined'); +ok(!defined($6), 'capture 6 is undefined'); + +ok(/((a?(c))(e))(d)/, 'equivalent required suffix pattern matches'); +is($1, 'ace', 'ordinary suffix pattern captures group 1'); +is($2, 'ac', 'ordinary suffix pattern captures group 2'); +is($3, 'c', 'ordinary suffix pattern captures group 3'); +is($4, 'e', 'ordinary suffix pattern captures group 4'); +is($5, 'd', 'ordinary suffix pattern captures group 5'); +ok(!defined($6), 'ordinary suffix pattern leaves group 6 undefined'); diff --git a/src/test/resources/unit/regex/callback_pos_state.t b/src/test/resources/unit/regex/callback_pos_state.t new file mode 100644 index 0000000000..3c0d587eb8 --- /dev/null +++ b/src/test/resources/unit/regex/callback_pos_state.t @@ -0,0 +1,64 @@ +use strict; +use warnings; +use Test::More tests => 24; + +my $message = 'pos inside (?{ })'; +my $str = 'abcde'; +our ($foo, $bar); +like($str, qr/b(?{$foo = $_; $bar = pos})c/, $message); +is($foo, $str, $message); +is($bar, 2, $message); +is(pos $str, undef, $message); + +undef $foo; +undef $bar; +pos $str = undef; +ok($str =~ /b(?{$foo = $_; $bar = pos})c/g, $message); +is($foo, $str, $message); +is($bar, 2, $message); +is(pos $str, 3, $message); + +$_ = $str; +undef $foo; +undef $bar; +like($_, qr/b(?{$foo = $_; $bar = pos})c/, $message); +is($foo, $str, $message); +is($bar, 2, $message); + +undef $foo; +undef $bar; +ok(/b(?{$foo = $_; $bar = pos})c/g, $message); +is($foo, $str, $message); +is($bar, 2, $message); +is(pos, 3, $message); + +undef $foo; +undef $bar; +pos = undef; +1 while /b(?{$foo = $_; $bar = pos})c/g; +is($foo, $str, $message); +is($bar, 2, $message); +is(pos, undef, $message); + +undef $foo; +undef $bar; +$_ = 'abcde|abcde'; +ok(s/b(?{$foo = $_; $bar = pos})c/x/g, $message); +is($foo, 'abcde|abcde', $message); +is($bar, 8, $message); +is($_, 'axde|axde', $message); + +our @res; +$_ = 'abcde|abcde'; +() = /([ace]).(?{push @res, $1,$2})([ce])(?{push @res, $1,$2})/g; +@res = map {defined $_ ? "'$_'" : 'undef'} @res; +is("@res", "'a' undef 'a' 'c' 'e' undef 'a' undef 'a' 'c'", $message); + +@res = (); +() = /([ace]).(?{push @res, $`,$&,$'})([ce])(?{push @res, $`,$&,$'})/g; +@res = map {defined $_ ? "'$_'" : 'undef'} @res; +is("@res", "'' 'ab' 'cde|abcde' " . + "'' 'abc' 'de|abcde' " . + "'abcd' 'e|' 'abcde' " . + "'abcde|' 'ab' 'cde' " . + "'abcde|' 'abc' 'de'", $message); diff --git a/src/test/resources/unit/regex/dynamic_code_array_interpolation.t b/src/test/resources/unit/regex/dynamic_code_array_interpolation.t new file mode 100644 index 0000000000..5e72f02155 --- /dev/null +++ b/src/test/resources/unit/regex/dynamic_code_array_interpolation.t @@ -0,0 +1,83 @@ +use strict; +use warnings; +use Test::More tests => 28; + +local $" = '-'; +my $pat; + +my $A = 'A'; +my $B = 'B'; +my $C = 'C'; +my $E = 'E'; +my $a = 'aa'; +my $b = 'bb'; +my $c = 'cc'; +my $e = 'ee'; + +my @r = (qr/(??{$B})/); + +like('B', qr/^@r$/, 'single code regex in array'); +like('AB', qr/^(??{$A})@r$/, 'literal code before array'); +like('XAB', qr/^X(??{$A})@r$/, 'literal and code before array'); +$pat = qr/^X(??{$A})@r(??{$C})$/; +like('XABC', $pat, 'code array between dynamic groups'); +unlike('', $pat, 'combined pattern rejects empty string'); +unlike('XAC', $pat, 'combined pattern requires array code'); +unlike('XAbbC', $pat, 'combined pattern rejects wrong array code'); + +{ + my $B = 'Q'; + push @r, qr/(??{$B})/; +} + +like('B-Q', qr/^@r$/, 'two code regexes in array'); +like('AB-Q', qr/^(??{$A})@r$/, 'literal code before two-item array'); +like('XAB-Q', qr/^X(??{$A})@r$/, 'literal before two-item array'); +$pat = qr/^X(??{$A})@r(??{$C})$/; +like('XAB-QC', $pat, 'two-item array between dynamic groups'); +unlike('', $pat, 'two-item pattern rejects empty string'); +unlike('XAC', $pat, 'two-item pattern requires array code'); +unlike('XAB-BC', $pat, 'two-item pattern preserves captured lexical'); + +package DynamicArrayLcConcat { + use overload + '""' => sub { ${$_[0]} }, + '.' => sub { + my ($x, $y) = @_[ $_[2] ? (1, 0) : (0, 1) ]; + my ($xx, $yy) = ("$x", "$y"); + lc("$xx=$yy"); + }; +} + +my $r = qr/(??{$E})/; +bless $r, 'DynamicArrayLcConcat'; + +use re 'eval'; + +like('=ee', qr/^$r$/, 'overloaded scalar code pattern'); +{ + no re 'eval'; + eval q{my $x = qr/^$r$/; 1}; + like($@, qr/Eval-group not allowed/, 'overloaded scalar needs re eval'); +} +like('aa=ee', qr/^(??{$A})$r$/, 'dynamic code before overloaded scalar'); +like('xaa=ee', qr/^X(??{$A})$r$/, 'literal before overloaded scalar'); +$pat = qr/^X(??{$A})$r(??{$C})$/; +like('xaa=eeC', $pat, 'overloaded scalar between dynamic groups'); +unlike('', $pat, 'overloaded scalar pattern rejects empty string'); +unlike('XA=EC', $pat, 'overloaded scalar preserves concatenation case'); + +push @r, $r; + +like('bb-bb-=ee', qr/^@r$/, 'overloaded scalar in code array'); +{ + no re 'eval'; + eval q{my $x = qr/^@r$/; 1}; + like($@, qr/Eval-group not allowed/, 'overloaded array needs re eval'); +} +like('aabb-bb-=ee', qr/^(??{$A})@r$/, 'dynamic code before overloaded array'); +like('xaabb-bb-=ee', qr/^X(??{$A})@r$/, 'literal before overloaded array'); +$pat = qr/^X(??{$A})@r(??{$C})$/; +like('xaabb-bb-=eeC', $pat, 'overloaded array between dynamic groups'); +unlike('', $pat, 'overloaded array pattern rejects empty string'); +unlike('XAB-B-=EC', $pat, 'overloaded array preserves concatenation case'); diff --git a/src/test/resources/unit/regex/invalid_unicode_property_diagnostics.t b/src/test/resources/unit/regex/invalid_unicode_property_diagnostics.t new file mode 100644 index 0000000000..074fa95dba --- /dev/null +++ b/src/test/resources/unit/regex/invalid_unicode_property_diagnostics.t @@ -0,0 +1,42 @@ +use strict; +use warnings; +use Test::More tests => 39; + +my $callback_calls = 0; +sub isfoo { $callback_calls++; return "0041" } +sub infoo { $callback_calls++; return "0041" } +sub ISfoo { $callback_calls++; return "0041" } +sub INfoo { $callback_calls++; return "0041" } +sub IsWhitespaceUpper { return "0041" } + +my @warnings; +local $SIG{__WARN__} = sub { push @warnings, @_ }; +local $ENV{JPERL_UNIMPLEMENTED} = 'warn'; + +for my $property (qw(q qrst f foo isfoo infoo ISfoo INfoo)) { + my $source = 'qr/\p{' . $property . '}/'; + my $ok = eval "$source; 1"; + ok(!$ok, "$property is rejected"); + like($@, qr/^Can't find Unicode property definition/, "$property reports the Perl diagnostic"); +} + +for my $source (qw(qr/\pq/ qr/\Pq/ qr/\pf/ qr/\Pf/)) { + my $ok = eval "$source; 1"; + ok(!$ok, "$source is rejected"); + like($@, qr/^Can't find Unicode property definition/, "$source reports the Perl diagnostic"); +} + +for my $property (qw(Is::foo In::foo main::Is::foo pkg::In::foo utf8::Is::foo)) { + my $source = 'qr/\p{' . $property . '}/'; + my $ok = eval "$source; 1"; + ok(!$ok, "$property is rejected"); + like($@, qr/^Illegal user-defined property name/, "$property reports an illegal name"); +} + +is($callback_calls, 0, 'noncanonical case variants do not call Perl property subs'); +is(scalar @warnings, 0, 'fatal property diagnostics are not downgraded to warnings'); + +my $spaced = qr/\p{ main::IsWhitespaceUpper }/; +ok($spaced, 'canonical user property permits surrounding whitespace'); +ok('A' =~ $spaced, 'spaced user property matches its definition'); +ok('a' !~ $spaced, 'spaced user property remains case-sensitive'); diff --git a/src/test/resources/unit/regex/last_successful_pattern.t b/src/test/resources/unit/regex/last_successful_pattern.t new file mode 100644 index 0000000000..c2a0c96e55 --- /dev/null +++ b/src/test/resources/unit/regex/last_successful_pattern.t @@ -0,0 +1,52 @@ +use strict; +use warnings; +use Test::More tests => 25; + +my ($str, $pat); +$str = 'ABCD'; +$str =~ /(D)/; +is("$1", 'D', '$1 is D'); +$pat = "${^LAST_SUCCESSFUL_PATTERN}"; +is($pat, '(?^:(D))', 'outer last successful pattern'); +{ + if ($str =~ /BX/ || $str =~ /(BC)/) { + is("$1", 'BC', '$1 is BC in inner scope'); + $pat = "${^LAST_SUCCESSFUL_PATTERN}"; + ok($str =~ s//ZZ/, 'empty pattern reuses inner pattern'); + is($str, 'AZZD', 'empty substitution used inner pattern'); + } +} +is("$1", 'D', '$1 restores after inner scope'); +is($pat, '(?^:(BC))', 'captured inner pattern string'); +ok($str =~ s//Q/, 'outer empty pattern substitution succeeds'); +is($str, 'AZZQ', 'outer empty pattern reuses outer pattern'); +$pat = "${^LAST_SUCCESSFUL_PATTERN}"; +is($pat, '(?^:(D))', 'outer last pattern restores'); + +$str = 'ABCD'; +{ + if ($str =~ /BX/ || $str =~ /(BC)/) { + is("$1", 'BC', '$1 is BC for explicit special pattern'); + $pat = "${^LAST_SUCCESSFUL_PATTERN}"; + ok($str =~ s/${^LAST_SUCCESSFUL_PATTERN}/ZZ/, + 'special variable matches as a pattern'); + is($str, 'AZZD', 'special variable substitution used inner pattern'); + } +} +is("$1", 'D', '$1 restores after explicit special pattern scope'); +is($pat, '(?^:(BC))', 'explicit inner pattern string'); +is($str, 'AZZD', 'explicit special pattern matches like empty pattern'); +ok($str =~ s/${^LAST_SUCCESSFUL_PATTERN}/Q/, + 'restored special pattern substitution succeeds'); +is($str, 'AZZQ', 'restored special pattern used outer pattern'); +ok($str =~ /ZQ/, 'new pattern matches'); +$pat = "${^LAST_SUCCESSFUL_PATTERN}"; +is($pat, '(?^:ZQ)', 'last successful pattern changes'); + +$str = 'foobarfoo'; +ok($str =~ s/foo//, 'matched foo'); +my $copy = ${^LAST_SUCCESSFUL_PATTERN}; +ok(defined($copy), 'copied pattern is defined'); +ok($str =~ s/bar//, 'matched bar'); +ok($str =~ s/$copy/PQR/, 'copied regex object remains usable'); +is($str, 'PQR', 'copied regex produces final value'); diff --git a/src/test/resources/unit/regex/reference_stringification.t b/src/test/resources/unit/regex/reference_stringification.t new file mode 100644 index 0000000000..cc30b17750 --- /dev/null +++ b/src/test/resources/unit/regex/reference_stringification.t @@ -0,0 +1,16 @@ +use strict; +use warnings; +use Test::More tests => 5; + +{ + package ReferenceStringificationS; + use overload '""' => sub { 'Object S' }; + sub new { bless [] } + + ::ok(do { \my $v } =~ /^SCALAR/, 'scalar reference stringification'); + ::ok(do { \\my $v } =~ /^REF/, 'reference-to-reference stringification'); + ::ok([] =~ /^ARRAY/, 'array reference stringification'); + ::ok({} =~ /^HASH/, 'hash reference stringification'); + ::ok('ReferenceStringificationS'->new =~ /^Object S/, + 'object stringification'); +} diff --git a/src/test/resources/unit/regex/runtime_control_verb_package.t b/src/test/resources/unit/regex/runtime_control_verb_package.t new file mode 100644 index 0000000000..9dbcde069b --- /dev/null +++ b/src/test/resources/unit/regex/runtime_control_verb_package.t @@ -0,0 +1,35 @@ +use strict; +use warnings; +use Test::More tests => 14; + +{ + package RuntimeControlVerbInactive; + our $REGERROR = 'inactive error'; + our $REGMARK = 'inactive mark'; +} + +{ + package RuntimeControlVerbPackage; + our $REGERROR; + local $REGERROR; + + for my $name ('', ':named') { + for my $verb ('PRUNE', 'SKIP', 'COMMIT') { + my $mark = $verb eq 'SKIP' && $name ? "(*MARK$name)" : ''; + my $pattern = "$mark(*$verb$name)(*FAIL)"; + 'aaaab' =~ /a+b$pattern/; + ::is($REGERROR, $name ? 'named' : '1', + "runtime $verb failure updates package REGERROR"); + + $pattern = "$mark(*$verb$name)"; + 'aaaab' =~ /a+b$pattern/; + ::is($REGERROR, '', + "runtime $verb success clears package REGERROR"); + } + } +} + +is($RuntimeControlVerbInactive::REGERROR, 'inactive error', + 'runtime control verbs do not update non-localized REGERROR variables'); +is($RuntimeControlVerbInactive::REGMARK, 'inactive mark', + 'runtime control verbs do not update non-localized REGMARK variables'); diff --git a/src/test/resources/unit/regex/runtime_dynamic_control_boundaries.t b/src/test/resources/unit/regex/runtime_dynamic_control_boundaries.t new file mode 100644 index 0000000000..0b74a30ee9 --- /dev/null +++ b/src/test/resources/unit/regex/runtime_dynamic_control_boundaries.t @@ -0,0 +1,33 @@ +use strict; +use warnings; +use re 'eval'; +use Test::More tests => 9; + +our ($count, $REGERROR); + +{ + local $REGERROR = 'sentinel'; + my $dynamic = 'a'; + 'a' =~ /(??{$dynamic})/; + is($REGERROR, 'sentinel', + 'dynamic patterns without control verbs leave REGERROR untouched'); +} + +for my $case ( + [PRUNE => 3], + [SKIP => 1], + [COMMIT => 1], + [THEN => 3], +) { + my ($verb, $expected_count) = @$case; + local $REGERROR; + $count = 0; + my $dynamic = "(*$verb)"; + + 'aaab' =~ /a+b?(??{$dynamic})(?{$count++})(*FAIL)/; + + is($count, $expected_count, + "dynamic $verb propagates its cut to the enclosing matcher"); + is($REGERROR, '1', + "dynamic $verb publishes its enclosing failure in REGERROR"); +} diff --git a/src/test/resources/unit/regex/unicode_pos_offsets.t b/src/test/resources/unit/regex/unicode_pos_offsets.t new file mode 100644 index 0000000000..7cfb44d32c --- /dev/null +++ b/src/test/resources/unit/regex/unicode_pos_offsets.t @@ -0,0 +1,32 @@ +use strict; +use warnings; +use Test::More tests => 11; + +my $smile = chr(0x1F642); +my $subject = ('A' . chr(0xE9) . $smile) x 2000; +my ($matches, $last_pos) = (0); +while ($subject =~ /./g) { + $matches++; + $last_pos = pos($subject); +} +is($matches, 6000, 'global match visits every Perl character'); +is($last_pos, 6000, 'successful global pos is in Perl characters'); +is(pos($subject), undef, 'terminal failed global match resets pos'); + +my $short = 'A' . $smile . 'B'; +pos($short) = 2; +ok($short =~ /\GB/g, 'global match converts pos beyond supplementary character'); +is(pos($short), 3, 'global match publishes logical end position'); + +pos($short) = 2; +ok($short =~ /\GB/, 'non-global G assertion converts assigned pos'); +is(pos($short), 2, 'successful non-global G assertion preserves pos'); + +my $replaced = 'A' . $smile . 'B'; +my $replacement_pos; +pos($replaced) = 2; +is($replaced =~ s/\GB/do { $replacement_pos = pos($replaced); 'X' }/e, 1, + 'substitution G assertion converts logical pos'); +is($replacement_pos, 2, 'replacement code observes logical match start'); +is($replaced, 'A' . $smile . 'X', 'substitution starts after supplementary character'); +is(pos($replaced), undef, 'destructive substitution invalidates pos'); diff --git a/third_party/joni/src/org/joni/ByteCodeMachine.java b/third_party/joni/src/org/joni/ByteCodeMachine.java index 87bf8163c3..ad1cb5fa9a 100644 --- a/third_party/joni/src/org/joni/ByteCodeMachine.java +++ b/third_party/joni/src/org/joni/ByteCodeMachine.java @@ -41,6 +41,11 @@ class ByteCodeMachine extends StackMachine implements MatchView { private static final int MAX_INTERRUPT_CHECK_EVERY = 256 << 7; // 32768 private static final int MAX_SUBEXP_CALL_DEPTH = 1000; + private static final int CONTROL_NONE = 0; + private static final int CONTROL_PRUNE = 1; + private static final int CONTROL_SKIP = 2; + private static final int CONTROL_THEN = 3; + private static final int CONTROL_COMMIT = 4; int interruptCheckEvery = 256; // << 1 after each check up to ^^^ volatile boolean interrupted = false; @@ -53,6 +58,7 @@ class ByteCodeMachine extends StackMachine implements MatchView { private int sbegin; private int pkeep; private int currentRegexOptions; + private int pendingControlAction; private final int[]code; // byte code private int ip; // instruction pointer @@ -2073,6 +2079,7 @@ private void opDynamicCallout() { DynamicContinuation continuation = new DynamicContinuation( nested, nestedHandler, s, range); int nestedEnd = continuation.first(); + continuation.propagateControlTo(this); if (nestedEnd < 0) { continuation.abort(); opFail(); @@ -2129,12 +2136,15 @@ private boolean opAccept() { } private void opPrune() { + controlVerbEncountered = true; String name = controlVerbName(code[ip++]); controlError = name == null ? "1" : name; cutAlternatives(false); + pendingControlAction = CONTROL_PRUNE; } private void opSkip() { + controlVerbEncountered = true; String name = controlVerbName(code[ip++]); controlError = name == null ? "1" : name; if (name != null) { @@ -2142,26 +2152,33 @@ private void opSkip() { if (target < 0) return; cutAlternatives(false); requestSearchSkip(target); + pendingControlAction = CONTROL_SKIP; return; } cutAlternatives(false); requestSearchSkip(s); + pendingControlAction = CONTROL_SKIP; } private void opThen() { + controlVerbEncountered = true; String name = controlVerbName(code[ip++]); controlError = name == null ? "1" : name; cutAlternatives(true, s, sprev, pkeep); + pendingControlAction = CONTROL_THEN; } private void opCommit() { + controlVerbEncountered = true; String name = controlVerbName(code[ip++]); controlError = name == null ? "1" : name; cutAlternatives(false); requestSearchAbort(); + pendingControlAction = CONTROL_COMMIT; } private void opMark() { + controlVerbEncountered = true; String next = controlVerbName(code[ip++]); pushControlMark(controlMark, next, s); controlMark = next; @@ -2383,6 +2400,7 @@ private void opFail() { int returnAddress = e.getStatePCode(); DynamicContinuation continuation = e.takeDynamicContinuation(); int nestedEnd = continuation.next(); + continuation.propagateControlTo(this); if (nestedEnd >= 0) { pushDynamicAlternative(returnAddress, continuation); ip = returnAddress; @@ -2465,6 +2483,35 @@ private int run() { } } + void propagateControlTo(ByteCodeMachine outer) { + int action = machine.pendingControlAction; + machine.pendingControlAction = CONTROL_NONE; + if (machine.controlVerbEncountered) outer.controlVerbEncountered = true; + if (machine.controlError != null) outer.controlError = machine.controlError; + switch (action) { + case CONTROL_PRUNE: + outer.cutAlternatives(false); + break; + case CONTROL_SKIP: + outer.cutAlternatives(false); + int target = machine.takeSearchSkipRequest(); + if (target >= 0) outer.requestSearchSkip(target); + break; + case CONTROL_THEN: + // A dynamic program is an alternation boundary in Perl. THEN + // cannot enter an enclosing branch, so crossing this boundary + // has the same outer-stack effect as PRUNE. + outer.cutAlternatives(false); + break; + case CONTROL_COMMIT: + outer.cutAlternatives(false); + if (machine.takeSearchAbortRequest()) outer.requestSearchAbort(); + break; + default: + break; + } + } + void complete() { if (closed) return; closed = true; diff --git a/third_party/joni/src/org/joni/Matcher.java b/third_party/joni/src/org/joni/Matcher.java index fe9caf918e..355dca5992 100644 --- a/third_party/joni/src/org/joni/Matcher.java +++ b/third_party/joni/src/org/joni/Matcher.java @@ -58,6 +58,7 @@ public abstract class Matcher extends IntHolder { private int skipSearchTo = -1; protected String controlMark; protected String controlError; + protected boolean controlVerbEncountered; private static final class SkipSearch extends RuntimeException { final int target; @@ -142,6 +143,10 @@ public final String getControlError() { return controlError; } + public final boolean hasEncounteredControlVerb() { + return controlVerbEncountered; + } + protected final void msaInit(int option, int start, int gpos) { msaOptions = option; msaStart = start; @@ -157,7 +162,20 @@ protected final void requestSearchAbort() { abortSearch = true; } + protected final int takeSearchSkipRequest() { + int target = skipSearchTo; + skipSearchTo = -1; + return target; + } + + protected final boolean takeSearchAbortRequest() { + boolean requested = abortSearch; + abortSearch = false; + return requested; + } + public final int match(int at, int range, int option) { + controlVerbEncountered = false; try { return matchCommon(at, range, option, false); } catch (InterruptedException ex) { @@ -166,6 +184,7 @@ public final int match(int at, int range, int option) { } public final int matchInterruptible(int at, int range, int option) throws InterruptedException { + controlVerbEncountered = false; return matchCommon(at, range, option, true); } @@ -369,6 +388,7 @@ private boolean matchCheck(int upperRange, int s, int prev, boolean interrupt) t } public final int search(int start, int range, int option) { + controlVerbEncountered = false; int next = start; while (true) { abortSearch = false; @@ -392,6 +412,7 @@ public final int search(int start, int range, int option) { } public final int search(int gpos, int start, int range, int option) { + controlVerbEncountered = false; try { return searchCommon(gpos, start, range, option, false); } catch (SkipSearch | AbortSearch control) { @@ -402,6 +423,7 @@ public final int search(int gpos, int start, int range, int option) { } public final int searchInterruptible(int start, int range, int option) throws InterruptedException { + controlVerbEncountered = false; int next = start; while (true) { abortSearch = false; @@ -423,6 +445,7 @@ public final int searchInterruptible(int start, int range, int option) throws In } public final int searchInterruptible(int gpos, int start, int range, int option) throws InterruptedException { + controlVerbEncountered = false; try { return searchCommon(gpos, start, range, option, true); } catch (SkipSearch | AbortSearch control) { diff --git a/third_party/joni/test/org/joni/test/TestCallout.java b/third_party/joni/test/org/joni/test/TestCallout.java index e29a64ff36..d3b8a9af29 100644 --- a/third_party/joni/test/org/joni/test/TestCallout.java +++ b/third_party/joni/test/org/joni/test/TestCallout.java @@ -365,6 +365,37 @@ public void unwind(Object token) { assertEquals(3, matcher.captureEnd(3)); } + @Test + public void dynamicControlVerbsPropagateToTheOuterMatcher() { + String[] verbs = {"PRUNE", "SKIP", "COMMIT", "THEN"}; + int[] expectedCalls = {3, 1, 1, 3}; + + for (int index = 0; index < verbs.length; index++) { + Regex outer = regex("a+b?(?{=DYNAMIC:1})(?{=CALL:2})(*FAIL)"); + Regex nested = regex("(*" + verbs[index] + ")"); + int[] calls = {0}; + CalloutHandler handler = new CalloutHandler() { + @Override + public CalloutResult execute(int id, MatchView match) { + calls[0]++; + return CalloutResult.CONTINUE; + } + + @Override + public DynamicPatternResult executeDynamic(int id, MatchView match) { + return new DynamicPatternResult(nested, null, null); + } + + @Override + public void unwind(Object token) { + } + }; + + assertEquals(-1, search(outer, "aaab", handler)); + assertEquals(verbs[index], expectedCalls[index], calls[0]); + } + } + private static CalloutHandler recordingHandler(List events, boolean fail) { return new CalloutHandler() { @Override From f99edadd70cfb0b6f50477ba73defe98b379a162 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:07:41 +0200 Subject: [PATCH 03/60] fix(regex): preserve callbacks in apostrophe-delimited regexes (#1007) Squashed integration of PR #1007 at 94cf0ffbf. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- .../frontend/parser/StringParser.java | 10 +++++-- .../regex/single_quote_literal_callback.t | 27 +++++++++++++++++++ 2 files changed, 35 insertions(+), 2 deletions(-) create mode 100644 src/test/resources/unit/regex/single_quote_literal_callback.t diff --git a/src/main/java/org/perlonjava/frontend/parser/StringParser.java b/src/main/java/org/perlonjava/frontend/parser/StringParser.java index e274ec91e6..d5dca10038 100644 --- a/src/main/java/org/perlonjava/frontend/parser/StringParser.java +++ b/src/main/java/org/perlonjava/frontend/parser/StringParser.java @@ -366,8 +366,14 @@ static Node parseRegexString(EmitterContext ctx, ParsedString rawStr, Parser par Node parsed; if (rawStr.startDelim == '\'') { - // single quote delimiter, use the string as-is - parsed = new StringNode(rawStr.buffers.getFirst(), rawStr.index); + // An apostrophe delimiter suppresses ordinary interpolation, but + // literal (?{...}) and (??{...}) blocks are still compiled as regex + // callbacks. Keep variable interpolation disabled while using the + // regex-aware parser so executable source retains its provenance. + parsed = StringDoubleQuoted.parseDoubleQuotedString( + ctx, rawStr, false, false, true, + parser != null ? parser.getHeredocNodes() : null, + null, true, isRegexQuoteConstruction); } else { // Check if /x modifier is present boolean hasXModifier = modifiers != null && modifiers.contains("x"); diff --git a/src/test/resources/unit/regex/single_quote_literal_callback.t b/src/test/resources/unit/regex/single_quote_literal_callback.t new file mode 100644 index 0000000000..380fd4b5ad --- /dev/null +++ b/src/test/resources/unit/regex/single_quote_literal_callback.t @@ -0,0 +1,27 @@ +use strict; +use warnings; +use Test::More tests => 10; + +our $out = 1; +ok('abc' =~ m'a(?{ $out = 2 })b', 'literal callback pattern matches'); +is($out, 2, 'literal callback executes without use re eval'); + +$out = 1; +ok(!('abc' =~ m'a(?{ $out = 3 })c'), 'failed literal callback branch does not match'); +is($out, 1, 'failed literal callback branch unwinds state'); + +my $name = 'expanded'; +ok('expanded' !~ m'^$name$', 'single-quote regex delimiter does not interpolate variables'); + +$out = 4; +ok('a' =~ m'a(?{ $out++ })', 'callback code still resolves variables'); +is($out, 5, 'callback variable update is visible'); + +ok('abc' =~ m'^a(??{"b"})c$', 'single-quote delimiter retains literal recursive callback'); + +my $quoted = qr'^$name$'; +ok('expanded' !~ $quoted, 'single-quote qr delimiter does not interpolate variables'); + +my ($search, $subject) = ('x', 'x'); +$subject =~ s'$search'y'; +is($subject, 'x', 'single-quote substitution pattern does not interpolate variables'); From 396f6c002c3f117cf36d62dfdc65728faec81daa Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:08:39 +0200 Subject: [PATCH 04/60] fix(regex): preserve overloaded regex callback provenance (#1010) Squashed integration of PR #1010 at 3c0189a5f. The duplicated PR #1007 change is already present as its own integration commit. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- .../runtime/regex/RuntimeRegexTemplate.java | 119 ++++++++++++++++-- .../unit/regex/overloaded_qr_provenance.t | 74 +++++++++++ 2 files changed, 183 insertions(+), 10 deletions(-) create mode 100644 src/test/resources/unit/regex/overloaded_qr_provenance.t diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegexTemplate.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegexTemplate.java index 45a81e94f5..c906928d78 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegexTemplate.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegexTemplate.java @@ -1,6 +1,10 @@ package org.perlonjava.runtime.regex; import org.perlonjava.runtime.operators.StringOperators; +import org.perlonjava.runtime.runtimetypes.Overload; +import org.perlonjava.runtime.runtimetypes.OverloadContext; +import org.perlonjava.runtime.runtimetypes.PerlCompilerException; +import org.perlonjava.runtime.runtimetypes.RuntimeArray; import org.perlonjava.runtime.runtimetypes.RuntimeBase; import org.perlonjava.runtime.runtimetypes.RuntimeList; import org.perlonjava.runtime.runtimetypes.RuntimeScalar; @@ -11,6 +15,8 @@ import java.util.regex.Matcher; import java.util.regex.Pattern; +import static org.perlonjava.runtime.runtimetypes.RuntimeScalarCache.scalarUndef; + /** Runtime interpolation result that keeps executable regex callbacks out of strings. */ public final class RuntimeRegexTemplate { private static final Pattern CALLOUT_ID = Pattern.compile( @@ -58,7 +64,10 @@ public static RuntimeScalar build(RuntimeList parts) { // A lone interpolation must retain its runtime type so qr // overloading and an already-compiled regex remain observable. // Only a parser-created callback needs a new template skeleton. - if (!(only.value instanceof RuntimeRegexCallback)) return only; + if (!(only.value instanceof RuntimeRegexCallback)) { + RuntimeScalar overloaded = resolveLoneRegexOverload(only); + return overloaded == null ? only : overloaded; + } } boolean hasBlessedPart = false; for (RuntimeBase part : parts.elements) { @@ -68,15 +77,10 @@ public static RuntimeScalar build(RuntimeList parts) { } } if (hasBlessedPart) { - RuntimeScalar concatenated = new RuntimeScalar(""); - for (RuntimeBase part : parts.elements) { - RuntimeScalar scalar = part.scalar(); - if (scalar.value instanceof RuntimeRegexCallback callback) { - scalar = new RuntimeScalar(callback.source == null ? "" : callback.source); - } - concatenated = StringOperators.stringConcat(concatenated, scalar); - } - return concatenated; + // Overload dispatch is complete after this pass. A dot overload is + // allowed to return another blessed value; assemble that result + // without dispatching the same interpolation a second time. + parts = resolveBlessedParts(parts); } StringBuilder pattern = new StringBuilder(); List callbacks = new ArrayList<>(); @@ -115,6 +119,101 @@ public static RuntimeScalar build(RuntimeList parts) { return result; } + private static RuntimeScalar resolveLoneRegexOverload(RuntimeScalar scalar) { + int blessId = RuntimeScalarType.blessedId(scalar); + if (blessId >= 0) return null; + OverloadContext overload = OverloadContext.prepare(blessId); + if (overload == null) return null; + + RuntimeScalar qr = resolveQrOverload(overload, scalar); + if (qr != null) return qr; + RuntimeScalar stringified = overload.tryOverload("(\"\"", unaryOverloadArguments(scalar)); + if (stringified != null) return resolveStringificationResult(stringified); + if (!overload.allowsFallbackAutogen()) { + throw new PerlCompilerException("Operation \"qr\": no method found,"); + } + return null; + } + + private static RuntimeList resolveBlessedParts(RuntimeList parts) { + RuntimeList resolved = new RuntimeList(); + resolved.add(new RuntimeScalar("")); + + for (RuntimeBase part : parts.elements) { + RuntimeScalar scalar = part.scalar(); + int blessId = RuntimeScalarType.blessedId(scalar); + if (blessId >= 0) { + resolved.add(scalar); + continue; + } + + OverloadContext overload = OverloadContext.prepare(blessId); + RuntimeScalar qr = overload == null ? null : resolveQrOverload(overload, scalar); + if (qr != null) { + resolved.add(qr); + continue; + } + + RuntimeScalar left = concatenateForOverload(resolved); + RuntimeScalar concatenated = OverloadContext.tryTwoArgumentOverloadDirect( + left, scalar, RuntimeScalarType.blessedId(left), blessId, "(."); + if (concatenated != null) { + resolved = new RuntimeList(concatenated); + continue; + } + + RuntimeScalar stringified = overload == null ? null + : overload.tryOverload("(\"\"", unaryOverloadArguments(scalar)); + if (stringified != null) { + resolved.add(resolveStringificationResult(stringified)); + continue; + } + + resolved = new RuntimeList(StringOperators.stringConcat(left, scalar)); + } + return resolved; + } + + /** + * A dot overload receives the textual pattern assembled so far. Executable + * callbacks on that side consequently become runtime source; only a qr or + * stringification overload that directly returns REGEXP keeps provenance. + */ + private static RuntimeScalar concatenateForOverload(RuntimeList parts) { + RuntimeScalar concatenated = new RuntimeScalar(""); + for (RuntimeBase part : parts.elements) { + RuntimeScalar scalar = part.scalar(); + if (scalar.value instanceof RuntimeRegexCallback callback) { + scalar = new RuntimeScalar(callback.source == null ? "" : callback.source); + } + concatenated = StringOperators.stringConcat(concatenated, scalar); + } + return concatenated; + } + + private static RuntimeScalar resolveQrOverload(OverloadContext overload, + RuntimeScalar scalar) { + RuntimeScalar qr = overload.tryOverload("(qr", new RuntimeArray(scalar)); + if (qr != null && qr.type != RuntimeScalarType.REGEX) { + throw new PerlCompilerException("Overloaded qr did not return a REGEXP"); + } + return qr; + } + + private static RuntimeScalar resolveStringificationResult(RuntimeScalar result) { + RuntimeScalar current = result; + for (int depth = 0; depth < 10 && RuntimeScalarType.blessedId(current) < 0; depth++) { + RuntimeScalar next = Overload.stringify(current); + if (next == current) break; + current = next; + } + return current; + } + + private static RuntimeArray unaryOverloadArguments(RuntimeScalar scalar) { + return new RuntimeArray(scalar, scalarUndef, new RuntimeScalar("")); + } + /** Preserve callback-bearing qr values while an array is joined for interpolation. */ public static RuntimeScalar buildJoined(RuntimeScalar separator, List elements) { diff --git a/src/test/resources/unit/regex/overloaded_qr_provenance.t b/src/test/resources/unit/regex/overloaded_qr_provenance.t new file mode 100644 index 0000000000..4fdbee3fc7 --- /dev/null +++ b/src/test/resources/unit/regex/overloaded_qr_provenance.t @@ -0,0 +1,74 @@ +use strict; +use warnings; +use Test::More tests => 13; + +{ + package Local::QrOverload; + use overload 'qr' => sub { qr/(??{50})/ }; +} + +{ + package Local::StringOverload; + use overload '""' => sub { qr/(??{51})/ }; +} + +{ + package Local::QrAndConcatOverload; + use overload + 'qr' => sub { qr/(??{50})/ }, + '.' => sub { $_[1] . qr/(??{60})/ }; +} + +{ + package Local::ConcatOverload; + use overload '.' => sub { $_[1] . qr/(??{52})/ }; +} + +{ + package Local::StringAndConcatOverload; + use overload + '""' => sub { qr/(??{7})/ }, + '.' => sub { $_[1] . qr/(??{53})/ }; +} + +{ + package Local::IndirectStringOverload; + use overload '""' => sub { $_[0][0] }; +} + +package main; + +my $qr_object = bless [], 'Local::QrOverload'; +ok('A50' =~ /^A$qr_object$/, 'embedded qr overload retains executable provenance'); +ok('50' =~ /$qr_object/, 'bare qr overload retains executable provenance'); + +my $string_object = bless [], 'Local::StringOverload'; +ok('A51' =~ /^A$string_object$/, 'embedded string overload returning qr retains provenance'); +ok('51' =~ /$string_object/, 'bare string overload returning qr retains provenance'); + +my $both_object = bless [], 'Local::QrAndConcatOverload'; +ok('A50' =~ /^A$both_object$/, 'embedded interpolation prefers qr overload to concat overload'); + +my $concat_object = bless [], 'Local::ConcatOverload'; +my $concat_error = eval { 'A52' =~ /^A$concat_object$/; 1 } ? '' : $@; +like($concat_error, qr/Eval-group not allowed/, 'concat overload source remains runtime source'); +my $bare_concat_error = eval { '52' =~ /$concat_object/; 1 } ? '' : $@; +like($bare_concat_error, qr/no method found/, 'bare interpolation does not use concat overload'); +{ + use re 'eval'; + ok('A52' =~ /^A$concat_object$/, 'use re eval permits concat overload source'); +} + +my $string_concat_object = bless [], 'Local::StringAndConcatOverload'; +my $string_concat_error = eval { 'A53' =~ /^A$string_concat_object$/; 1 } ? '' : $@; +like($string_concat_error, qr/Eval-group not allowed/, 'concat overload precedes string overload'); +my $bare_string_error = eval { '7' =~ /$string_concat_object/; 1 } ? '' : $@; +is($bare_string_error, '', 'bare interpolation uses string overload without concat'); +{ + use re 'eval'; + ok('A53' =~ /^A$string_concat_object$/, 'use re eval permits preferred concat overload source'); +} + +my $indirect_object = bless [ $string_object ], 'Local::IndirectStringOverload'; +ok('A51' =~ /^A$indirect_object/, 'indirect string overload retains executable provenance'); +ok('51' =~ /$indirect_object/, 'bare indirect string overload retains provenance'); From 5bc62633354eb4cf8fb2530b74960b7aab5b65ae Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:09:46 +0200 Subject: [PATCH 05/60] perf(regex): reuse Joni subject offset tables (#1008) Squashed integration of PR #1008 at a42ed9128. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- .../runtime/regex/JoniRegexPattern.java | 39 +++++++++++++++---- .../runtime/regex/JoniRegexPatternTest.java | 25 ++++++++++++ .../unit/regex/joni_global_offset_reuse.t | 16 ++++++++ 3 files changed, 72 insertions(+), 8 deletions(-) create mode 100644 src/test/resources/unit/regex/joni_global_offset_reuse.t diff --git a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java index 6d96310ece..2b23a6fe28 100644 --- a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java +++ b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java @@ -21,6 +21,7 @@ import java.util.LinkedHashMap; import java.util.Map; import java.util.List; +import java.util.WeakHashMap; import org.perlonjava.runtime.operators.WarnDie; import org.perlonjava.runtime.runtimetypes.*; @@ -30,6 +31,8 @@ * beyond the Java Pattern fast path. */ final class JoniRegexPattern { + private static final Map INPUT_ENCODINGS = new WeakHashMap<>(); + // Ruby syntax defaults \w to ASCII even for a Unicode encoding. Perl's // default and /u modes use Unicode character classes; /a adds ASCII_RANGE // explicitly in toJoniOptions(). Keep the richer Ruby parser surface used @@ -81,6 +84,21 @@ RegexMatcher matcher(String input, List callbacks, hasControlVerbState, input, callbacks, subject); } + record InputEncoding(byte[] bytes, int[] charToByte, int[] byteToChar) {} + + static InputEncoding inputEncoding(String input) { + synchronized (INPUT_ENCODINGS) { + return INPUT_ENCODINGS.computeIfAbsent(input, JoniRegexPattern::buildInputEncoding); + } + } + + private static InputEncoding buildInputEncoding(String input) { + byte[] bytes = input.getBytes(StandardCharsets.UTF_8); + int[] charToByte = JoniRegexMatcher.buildCharToByte(input); + int[] byteToChar = JoniRegexMatcher.buildByteToChar(input, bytes.length, charToByte); + return new InputEncoding(bytes, charToByte, byteToChar); + } + String patternDescription() { return sourcePattern; } @@ -602,9 +620,10 @@ private static final class JoniRegexMatcher implements RegexMatcher { this.input = input; this.callbacks = callbacks; this.subject = subject; - this.bytes = input.getBytes(StandardCharsets.UTF_8); - this.charToByte = buildCharToByte(input); - this.byteToChar = buildByteToChar(input, bytes.length, charToByte); + InputEncoding encoding = inputEncoding(input); + this.bytes = encoding.bytes(); + this.charToByte = encoding.charToByte(); + this.byteToChar = encoding.byteToChar(); region(0, input.length()); } @@ -772,12 +791,16 @@ private static int[] buildCharToByte(String input) { private static int[] buildByteToChar(String input, int byteLength, int[] charToByte) { int[] offsets = new int[byteLength + 1]; - int charOffset = 0; - for (int b = 0; b <= byteLength; b++) { - while (charOffset + 1 < charToByte.length && charToByte[charOffset + 1] <= b) { - charOffset++; + for (int charOffset = 0; charOffset < input.length();) { + int chars = Character.charCount(input.codePointAt(charOffset)); + int nextCharOffset = charOffset + chars; + int nextByteOffset = charToByte[nextCharOffset]; + for (int byteOffset = charToByte[charOffset]; + byteOffset < nextByteOffset; byteOffset++) { + offsets[byteOffset] = charOffset; } - offsets[b] = charOffset; + offsets[nextByteOffset] = nextCharOffset; + charOffset = nextCharOffset; } return offsets; } diff --git a/src/test/java/org/perlonjava/runtime/regex/JoniRegexPatternTest.java b/src/test/java/org/perlonjava/runtime/regex/JoniRegexPatternTest.java index 61a7d4862f..8029ac42f8 100644 --- a/src/test/java/org/perlonjava/runtime/regex/JoniRegexPatternTest.java +++ b/src/test/java/org/perlonjava/runtime/regex/JoniRegexPatternTest.java @@ -5,6 +5,8 @@ import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertSame; import static org.junit.jupiter.api.Assertions.assertTrue; @Tag("unit") @@ -94,4 +96,27 @@ void acceptsWhitespaceInsidePerlIntervals() { assertTrue(new JoniRegexPattern("\\p{Latin}{ , 2 }", FLAGS) .matcher("a", java.util.List.of()).find()); } + + @Test + void reusesImmutableInputEncodingAndPreservesSupplementaryOffsets() { + String input = new String("A\u00E9\uD83D\uDE42Z"); + + JoniRegexPattern.InputEncoding first = JoniRegexPattern.inputEncoding(input); + JoniRegexPattern.InputEncoding second = JoniRegexPattern.inputEncoding(input); + + assertSame(first, second); + assertArrayEquals(new int[] {0, 1, 3, 3, 7, 8}, first.charToByte()); + assertArrayEquals(new int[] {0, 1, 1, 2, 2, 2, 2, 4, 5}, first.byteToChar()); + } + + @Test + void supplementaryCaptureUsesTheHighSurrogateBoundary() { + RegexMatcher matcher = new JoniRegexPattern("(.)", FLAGS) + .matcher("\uD83D\uDE42", java.util.List.of()); + + assertTrue(matcher.find()); + assertEquals("\uD83D\uDE42", matcher.group(1)); + assertEquals(0, matcher.start(1)); + assertEquals(2, matcher.end(1)); + } } diff --git a/src/test/resources/unit/regex/joni_global_offset_reuse.t b/src/test/resources/unit/regex/joni_global_offset_reuse.t new file mode 100644 index 0000000000..decfb37b93 --- /dev/null +++ b/src/test/resources/unit/regex/joni_global_offset_reuse.t @@ -0,0 +1,16 @@ +use strict; +use warnings; +use Test::More tests => 3; + +my $repetitions = 2_000; +my $subject = "A\x{E9}\x{1F642}" x $repetitions; +my ($matches, $last) = (0, ''); + +while ($subject =~ /(.)/g) { + $matches++; + $last = $1; +} + +is($matches, 3 * $repetitions, 'global match visits every Unicode code point'); +is($last, "\x{1F642}", 'last capture retains a supplementary code point'); +ok(!defined(pos($subject)), 'failed terminal global match resets pos'); From 6621778e150e44bd001b52d5d83c103f050fdfcf Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:09:46 +0200 Subject: [PATCH 06/60] fix(regex): preserve properties inside extended regex sets (#1009) Squashed integration of PR #1009 at 60bd283bd. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- .../runtime/regex/JoniRegexPattern.java | 29 ++++++++++++++++++- .../runtime/regex/JoniRegexPatternTest.java | 11 +++++++ .../unit/regex/regex_sets_ascii_property.t | 9 ++++++ 3 files changed, 48 insertions(+), 1 deletion(-) create mode 100644 src/test/resources/unit/regex/regex_sets_ascii_property.t diff --git a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java index 2b23a6fe28..c7d587428d 100644 --- a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java +++ b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java @@ -117,6 +117,7 @@ private static UserPropertyTranslation translateUserDefinedProperties( String pattern, RegexFlags flags) { StringBuilder translated = new StringBuilder(pattern.length()); boolean deferred = false; + int extendedClassBracketDepth = 0; for (int i = 0; i < pattern.length(); i++) { char ch = pattern.charAt(i); if (ch == '\\' && i + 1 < pattern.length() && pattern.charAt(i + 1) == '\\') { @@ -124,10 +125,31 @@ private static UserPropertyTranslation translateUserDefinedProperties( i++; continue; } + if (extendedClassBracketDepth == 0 && pattern.startsWith("(?[", i)) { + translated.append("(?["); + extendedClassBracketDepth = 1; + i += 2; + continue; + } + if (extendedClassBracketDepth > 0 && ch == '[') { + extendedClassBracketDepth++; + translated.append(ch); + continue; + } + if (extendedClassBracketDepth > 0 && ch == ']') { + extendedClassBracketDepth--; + translated.append(ch); + continue; + } if (ch != '\\' || i + 3 >= pattern.length() || (pattern.charAt(i + 1) != 'p' && pattern.charAt(i + 1) != 'P') || pattern.charAt(i + 2) != '{') { - translated.append(ch); + if (ch == '\\' && i + 1 < pattern.length()) { + translated.append(pattern, i, i + 2); + i++; + } else { + translated.append(ch); + } continue; } int end = pattern.indexOf('}', i + 3); @@ -143,6 +165,11 @@ private static UserPropertyTranslation translateUserDefinedProperties( "(?i)^(?:scx|script[_ ]?extensions)\\s*=.*"); boolean frontendProperty = unnegated.matches( "(?i)^(?:script|block|blk|age|in|present[_ ]?in)\\s*=.*"); + if (frontendProperty && extendedClassBracketDepth > 0) { + translated.append(pattern, i, end + 1); + i = end; + continue; + } if (!userDefined && !scriptExtensions && !frontendProperty) { translated.append(pattern, i, end + 1); i = end; diff --git a/src/test/java/org/perlonjava/runtime/regex/JoniRegexPatternTest.java b/src/test/java/org/perlonjava/runtime/regex/JoniRegexPatternTest.java index 8029ac42f8..7d030eaf34 100644 --- a/src/test/java/org/perlonjava/runtime/regex/JoniRegexPatternTest.java +++ b/src/test/java/org/perlonjava/runtime/regex/JoniRegexPatternTest.java @@ -119,4 +119,15 @@ void supplementaryCaptureUsesTheHighSurrogateBoundary() { assertEquals(0, matcher.start(1)); assertEquals(2, matcher.end(1)); } + + @Test + void resolvesBlockPropertiesInsideExtendedClassesBeforeJoniCompilation() { + JoniRegexPattern pattern = new JoniRegexPattern( + "(?[ [k] + \\p{Blk=ASCII} ])", + RegexFlags.fromModifiers("i", "(?[ [k] + \\p{Blk=ASCII} ])")); + + assertTrue(pattern.matcher("k", java.util.List.of()).find()); + assertTrue(pattern.matcher("A", java.util.List.of()).find()); + assertFalse(pattern.matcher("\u017F", java.util.List.of()).find()); + } } diff --git a/src/test/resources/unit/regex/regex_sets_ascii_property.t b/src/test/resources/unit/regex/regex_sets_ascii_property.t new file mode 100644 index 0000000000..2554cd107c --- /dev/null +++ b/src/test/resources/unit/regex/regex_sets_ascii_property.t @@ -0,0 +1,9 @@ +use strict; +use warnings; +use Test::More tests => 3; + +my $pattern = qr/(?[ [k] + \p{Blk=ASCII} ])/i; + +ok("k" =~ $pattern, 'case-folded literal remains in the set'); +ok("A" =~ $pattern, 'ASCII block property remains in the set'); +ok("\x{17f}" !~ $pattern, '/i does not expand the ASCII block property'); From 829582ffe43c36fe5e63ab623e7107da262eb5cd Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:09:46 +0200 Subject: [PATCH 07/60] fix(regex): enable safe Joni backtracking checks (#1012) Squashed integration of PR #1012 at b80a5378b. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- third_party/joni/src/org/joni/Analyser.java | 4 +++- third_party/joni/src/org/joni/ArrayCompiler.java | 7 ++++--- third_party/joni/src/org/joni/ByteCodeMachine.java | 4 ---- third_party/joni/src/org/joni/Compiler.java | 7 ++++--- third_party/joni/src/org/joni/Config.java | 2 +- third_party/joni/src/org/joni/Matcher.java | 2 +- third_party/joni/src/org/joni/StackMachine.java | 2 +- third_party/joni/test/org/joni/test/TestPerl.java | 6 ++++++ 8 files changed, 20 insertions(+), 14 deletions(-) diff --git a/third_party/joni/src/org/joni/Analyser.java b/third_party/joni/src/org/joni/Analyser.java index acb3691fbc..c588d495d3 100644 --- a/third_party/joni/src/org/joni/Analyser.java +++ b/third_party/joni/src/org/joni/Analyser.java @@ -130,7 +130,9 @@ protected final void compile() { } if (Config.USE_CEC) { - if (env.backrefedMem == 0 || (Config.USE_SUBEXP_CALL && env.numCall == 0)) { + if (env.backrefedMem == 0 && + (!Config.USE_SUBEXP_CALL || env.numCall == 0) && + !env.hasCallout) { setupCombExpCheck(root, 0); if (Config.USE_SUBEXP_CALL && env.hasRecursion) { diff --git a/third_party/joni/src/org/joni/ArrayCompiler.java b/third_party/joni/src/org/joni/ArrayCompiler.java index 4aed58b758..d4d665e692 100644 --- a/third_party/joni/src/org/joni/ArrayCompiler.java +++ b/third_party/joni/src/org/joni/ArrayCompiler.java @@ -1355,10 +1355,11 @@ private int compileLengthTree(Node node) { break; case NodeType.QTFR: - if (Config.USE_CEC) { - len = compileCECLengthQuantifierNode((QuantifierNode)node); + QuantifierNode quantifier = (QuantifierNode)node; + if (Config.USE_CEC && regex.numCombExpCheck > 0 && quantifier.combExpCheckNum > 0) { + len = compileCECLengthQuantifierNode(quantifier); } else { - len = compileNonCECLengthQuantifierNode((QuantifierNode)node); + len = compileNonCECLengthQuantifierNode(quantifier); } break; diff --git a/third_party/joni/src/org/joni/ByteCodeMachine.java b/third_party/joni/src/org/joni/ByteCodeMachine.java index ad1cb5fa9a..e69872013c 100644 --- a/third_party/joni/src/org/joni/ByteCodeMachine.java +++ b/third_party/joni/src/org/joni/ByteCodeMachine.java @@ -1151,7 +1151,6 @@ private void opStateCheckAnyCharStar() { sprev = s; s += n; } - sprev = sbegin; // break; } private void opStateCheckAnyCharStarSb() { @@ -1165,7 +1164,6 @@ private void opStateCheckAnyCharStarSb() { sprev = s; s++; } - sprev = sbegin; // break; } // CEC @@ -1181,7 +1179,6 @@ private void opStateCheckAnyCharMLStar() { sprev = s; s += n; } - sprev = sbegin; // break; } private void opStateCheckAnyCharMLStarSb() { @@ -1193,7 +1190,6 @@ private void opStateCheckAnyCharMLStarSb() { sprev = s; s++; } - sprev = sbegin; // break; } private void opWord() { diff --git a/third_party/joni/src/org/joni/Compiler.java b/third_party/joni/src/org/joni/Compiler.java index 145eebdc32..c41ed626bb 100644 --- a/third_party/joni/src/org/joni/Compiler.java +++ b/third_party/joni/src/org/joni/Compiler.java @@ -162,10 +162,11 @@ protected final void compileTree(Node node) { break; case NodeType.QTFR: - if (Config.USE_CEC) { - compileCECQuantifierNode((QuantifierNode)node); + QuantifierNode quantifier = (QuantifierNode)node; + if (Config.USE_CEC && regex.numCombExpCheck > 0 && quantifier.combExpCheckNum > 0) { + compileCECQuantifierNode(quantifier); } else { - compileNonCECQuantifierNode((QuantifierNode)node); + compileNonCECQuantifierNode(quantifier); } break; diff --git a/third_party/joni/src/org/joni/Config.java b/third_party/joni/src/org/joni/Config.java index 90eec0ba20..849e4fe085 100644 --- a/third_party/joni/src/org/joni/Config.java +++ b/third_party/joni/src/org/joni/Config.java @@ -44,7 +44,7 @@ public interface Config extends org.jcodings.Config { boolean USE_WORD_BEGIN_END = ConfigSupport.getBoolean("joni.use_word_begin_end", true); /* "\<": word-begin, "\>": word-end */ boolean USE_FIND_LONGEST_SEARCH_ALL_OF_RANGE = ConfigSupport.getBoolean("joni.use_find_longest_search_all_of_range", true); boolean USE_SUNDAY_QUICK_SEARCH = ConfigSupport.getBoolean("joni.use_sunday_quick_search", true); - boolean USE_CEC = ConfigSupport.getBoolean("joni.use_cec", false); + boolean USE_CEC = ConfigSupport.getBoolean("joni.use_cec", true); boolean USE_DYNAMIC_OPTION = ConfigSupport.getBoolean("joni.use_dynamic_option", false); boolean USE_BYTE_MAP = ConfigSupport.getBoolean("joni.use_byte_map", OptExactInfo.OPT_EXACT_MAXLEN <= CHAR_TABLE_SIZE); boolean USE_INT_MAP_BACKWARD = ConfigSupport.getBoolean("joni.use_int_map_backward", false); diff --git a/third_party/joni/src/org/joni/Matcher.java b/third_party/joni/src/org/joni/Matcher.java index 355dca5992..a42e8473c7 100644 --- a/third_party/joni/src/org/joni/Matcher.java +++ b/third_party/joni/src/org/joni/Matcher.java @@ -192,7 +192,7 @@ private final int matchCommon(int at, int range, int option, boolean interrupt) msaInit(option, at, at); if (Config.USE_CEC) { - int offset = at = str; + int offset = at - str; stateCheckBuffInit(end - str, offset, regex.numCombExpCheck); // move it to construction? } // USE_COMBINATION_EXPLOSION_CHECK diff --git a/third_party/joni/src/org/joni/StackMachine.java b/third_party/joni/src/org/joni/StackMachine.java index bdaba8b37a..c0e13eb50f 100644 --- a/third_party/joni/src/org/joni/StackMachine.java +++ b/third_party/joni/src/org/joni/StackMachine.java @@ -150,7 +150,7 @@ protected final void stateCheckBuffInit(int strLength, int offset, int stateNum) // same impl, reduce... stateCheckBuff = new byte[size]; } - Arrays.fill(stateCheckBuff, offset, (size - offset), (byte)0); + Arrays.fill(stateCheckBuff, offset, size, (byte)0); stateCheckBuffSize = size; } else { stateCheckBuff = null; // reduce diff --git a/third_party/joni/test/org/joni/test/TestPerl.java b/third_party/joni/test/org/joni/test/TestPerl.java index 98bbf09f84..dd27926c11 100755 --- a/third_party/joni/test/org/joni/test/TestPerl.java +++ b/third_party/joni/test/org/joni/test/TestPerl.java @@ -44,4 +44,10 @@ public Syntax syntax() { @Override public void test() throws Exception { } + + @org.junit.Test(timeout = 5000) + public void testNestedQuantifierCombinationExplosion() throws Exception { + x2s(".X(.+)+X", "bXcXaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0, 4); + org.junit.Assert.assertEquals(0, nfail + nerror); + } } From 2f49b76d792623d8edcee3946d20f930a446c8e0 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:10:23 +0200 Subject: [PATCH 08/60] fix(regex): match user Unicode property diagnostics (#1011) Squashed integration of PR #1011 at 7410cabcf. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 25 +++- .../runtime/regex/RuntimeRegex.java | 2 +- .../runtime/regex/UnicodeResolver.java | 117 ++++++++++++------ .../user_property_definition_diagnostics.t | 34 +++++ 4 files changed, 130 insertions(+), 48 deletions(-) create mode 100644 src/test/resources/unit/regex/user_property_definition_diagnostics.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index e08971e59d..3b1d2767cf 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -230,6 +230,12 @@ Perl's user-property callback path. A 39-assertion standard-Perl oracle passes on JVM and interpreter; the unchanged `regexp_unicode_prop.t` runner improves from 1,039/1,110 to 1,054/1,110, leaving built-in aliases and six user-property-definition diagnostic assertions as separate work. +Direct user-property definitions now report Perl-compatible overflow, reversed +range, invalid component, callback death, recursion-chain, and package-name +diagnostics. The focused 12-assertion oracle passes on both execution backends; +`regexp_unicode_prop.t` gains two more assertions. Four deferred bare-name +assertions retain correct diagnostic content but still need literal source-order +provenance to add the canonical `main::` prefix. Executable callback source and literal trailing `/x` comments survive canonical regex-object stringification on both execution backends. Recursive Joni call @@ -350,6 +356,11 @@ matcher-specific timeouts on both execution backends. naming convention and made unknown-property diagnostics fatal even in compatibility warning mode. The focused oracle passes 39/39 on system Perl, JVM, and interpreter; `regexp_unicode_prop.t` gains 15 assertions. + - [x] Matched user-property definition validation, deterministic recursion + chains, callback-death wrapping, and direct package-name policy. + The focused oracle passes 12/12 on system Perl, JVM, and interpreter; + unchanged upstream coverage gains two assertions, with four source-order + provenance assertions tracked separately. - [ ] Complete the remaining Unicode aliases and diagnostic parity inventory. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) @@ -366,16 +377,18 @@ matcher-specific timeouts on both execution backends. ### Next Steps -1. Remediate the seven reduced forced-Joni failure causes, starting with the - catastrophic `regexp*` cluster and the now-profiled `pat_psycho*` matcher - reconstruction path; keep per-child hard limits throughout. +1. Remediate the remaining forced-Joni failure causes, starting with the + catastrophic `regexp*` cluster and absent generated Unicode fixtures; keep + per-child hard limits throughout. PR #1008 has removed the release-blocking + `pat_psycho*` offset-map reconstruction and bounded profiling confirms that + matcher-wrapper reuse is no longer on the critical path. 2. Integrate the stacked Joni offset-map, regex-set property, and CEC timeout slices, then rerun the forced-Joni corpus from the new combined head. 3. Capture the clean-branch JVM and interpreter 80-file baselines and compare both to PR 958 with the regression exit gate. -4. Integrate the exact invalid-property dispatch/diagnostic slice, then finish - PerlOnJava4's built-in Unicode-alias slice and the six residual - user-property-definition diagnostic assertions against Perl 5.44 behavior. +4. Integrate the invalid-property and definition-diagnostic slices, then finish + PerlOnJava4's built-in Unicode aliases and preserve compile-time source-order + provenance for the four deferred bare-name diagnostics. 5. Keep the warning-free whole-unit-suite gate green with Joni as the default; use explicit Java mode only to classify corpus regressions before Phase 5 removes the legacy backend and selector. diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java index 77cf10b877..63c3c9d87d 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java @@ -901,7 +901,7 @@ private static RuntimeRegex ensureCompiledForRuntime(RuntimeRegex regex) { // User property subs can execute arbitrary Perl and block. Resolve them // before compile() takes its process-wide monitor; only simultaneous // definitions of the same property coordinate in PerlThreadRegistry. - UnicodeResolver.preloadUserDefinedProperties(regex.patternString, + UnicodeResolver.preloadDeferredUserDefinedProperties(regex.patternString, regex.regexFlags != null && regex.regexFlags.isCaseInsensitive()); RuntimeRegex recompiled = compile(regex.patternString, regex.regexFlags == null ? "" : regex.regexFlags.toFlagString(), diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index 1b92bdd941..0942dc00a7 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -6,7 +6,10 @@ import org.perlonjava.app.scriptengine.PerlLanguageProvider; import org.perlonjava.runtime.runtimetypes.*; -import java.util.HashSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.LinkedHashSet; +import java.util.List; import java.util.Map; import java.util.Set; import java.util.regex.Pattern; @@ -199,15 +202,16 @@ private static String parseUserDefinedProperty(String definition, Set re } } else { // Parse hex range - extract the hex part before any comments - String hexPart = line.split("#")[0].trim(); - // Split by tabs or multiple spaces - String[] parts = hexPart.split("\\t+|\\s{2,}"); + String hexPart = stripDefinitionComment(line); + String[] parts = hexPart.split("\\s+"); if (parts.length == 1 && !parts[0].isEmpty()) { // Single character String hexStr = parts[0].trim(); // Check if it's a valid hex string if (!hexStr.matches("[0-9A-Fa-f]+")) { - throw new IllegalArgumentException("Can't find Unicode property definition \"" + line.trim() + "\" in expansion of " + propertyName); + throw new IllegalArgumentException( + "Can't find Unicode property definition \"" + hexPart + + "\" in expansion of " + propertyName); } try { long codePoint = Long.parseLong(hexStr, 16); @@ -218,7 +222,9 @@ private static String parseUserDefinedProperty(String definition, Set re } resultSet.add((int) codePoint); } catch (NumberFormatException e) { - throw new IllegalArgumentException("Can't find Unicode property definition \"" + line.trim() + "\" in expansion of " + propertyName); + throw new IllegalArgumentException( + "Code point too large in \"" + line + + "\" in expansion of " + propertyName); } } else if (parts.length >= 2) { // Range @@ -227,7 +233,9 @@ private static String parseUserDefinedProperty(String definition, Set re // Check if they're valid hex strings if (!startHex.matches("[0-9A-Fa-f]+") || !endHex.matches("[0-9A-Fa-f]+")) { - throw new IllegalArgumentException("Can't find Unicode property definition \"" + line.trim() + "\" in expansion of " + propertyName); + throw new IllegalArgumentException( + "Can't find Unicode property definition \"" + hexPart + + "\" in expansion of " + propertyName); } try { @@ -249,7 +257,9 @@ private static String parseUserDefinedProperty(String definition, Set re resultSet.add((int) start, (int) end); } catch (NumberFormatException e) { - throw new IllegalArgumentException("Can't find Unicode property definition \"" + line.trim() + "\" in expansion of " + propertyName); + throw new IllegalArgumentException( + "Code point too large in \"" + line + + "\" in expansion of " + propertyName); } } } @@ -287,6 +297,16 @@ private static boolean removeHexRange(UnicodeSet target, String definition) { } } + private static IllegalArgumentException recursivePropertyError( + String property, Set recursionSet) { + List expansionChain = new ArrayList<>(recursionSet); + Collections.reverse(expansionChain); + return new IllegalArgumentException( + "Infinite recursion in user-defined property \"" + property + + "\" in expansion of " + + String.join(" in expansion of ", expansionChain)); + } + /** * Resolves a property reference to a UnicodeSet (like utf8::InHiragana or main::IsMyProp). * Returns a UnicodeSet directly instead of a Java regex pattern string, so the result @@ -301,18 +321,7 @@ private static UnicodeSet resolvePropertyReferenceAsSet(String propRef, Set 0) { - chain.append(" in expansion of "); - } - chain.append(prop); - } - if (chain.length() > 0) { - chain.append(" in expansion of "); - } - chain.append(propRef); - throw new IllegalArgumentException("Infinite recursion in user-defined property \"" + propRef + "\" in expansion of " + chain); + throw recursivePropertyError(propRef, recursionSet); } // Remove utf8:: prefix if present @@ -483,6 +492,12 @@ private static UnicodeSet resolveStandardPropertyAsSet(String property, Set recursionSet, boolean caseInsensitive) { + return tryUserDefinedProperty(property, recursionSet, caseInsensitive, false); + } + + private static String tryUserDefinedProperty( + String property, Set recursionSet, boolean caseInsensitive, + boolean qualifyBareDiagnosticName) { // Build the full subroutine name String subName = property; if (!subName.contains("::")) { @@ -495,18 +510,9 @@ private static String tryUserDefinedProperty( // namespace; otherwise InFoo -> main::InFoo can wait on its own active // future instead of reporting recursive expansion. if (recursionSet.contains(subName)) { - StringBuilder chain = new StringBuilder(); - for (String recursiveProperty : recursionSet) { - if (chain.length() > 0) chain.append(" in expansion of "); - chain.append(recursiveProperty); - } - if (chain.length() > 0) chain.append(" in expansion of "); - chain.append(subName); - throw new IllegalArgumentException( - "Infinite recursion in user-defined property \"" + subName - + "\" in expansion of " + chain); + throw recursivePropertyError(subName, recursionSet); } - Set newRecursionSet = new HashSet<>(recursionSet); + Set newRecursionSet = new LinkedHashSet<>(recursionSet); newRecursionSet.add(subName); // Check cache first — Perl only calls user-defined property subs once @@ -531,20 +537,26 @@ private static String tryUserDefinedProperty( RuntimeScalar codeRef = GlobalVariable.getGlobalCodeRef(subName); final String resolvedSubName = subName; + final String diagnosticName = qualifyBareDiagnosticName + && !property.contains("::") ? subName : property; final String coordinationKey = cacheKey; try { String parsed = PerlRuntime.current().threadRegistry() .resolveUserUnicodeProperty(coordinationKey, () -> resolveUserDefinedProperty( - codeRef, resolvedSubName, newRecursionSet, + codeRef, resolvedSubName, diagnosticName, + newRecursionSet, caseInsensitive)); userPropertyCache().put(cacheKey, parsed); return parsed; + } catch (PerlDieException e) { + throw propertyDefinitionDie(e, diagnosticName); } catch (PerlCompilerException e) { // Re-throw Perl exceptions (like die in IsDeath) String msg = e.getMessage(); if (msg != null && !msg.contains("in expansion of")) { - throw new IllegalArgumentException("Died" + (msg.isEmpty() ? "" : ": " + msg) + " in expansion of " + subName, e); + throw new IllegalArgumentException("Died" + (msg.isEmpty() ? "" : ": " + msg) + + " in expansion of " + diagnosticName, e); } throw e; } catch (IllegalArgumentException e) { @@ -556,9 +568,9 @@ private static String tryUserDefinedProperty( } } - private static String resolveUserDefinedProperty(RuntimeScalar codeRef, String subName, - Set recursionSet, - boolean caseInsensitive) { + private static String resolveUserDefinedProperty( + RuntimeScalar codeRef, String subName, String diagnosticName, + Set recursionSet, boolean caseInsensitive) { try { // Perl passes one false/true argument for case-sensitive/folded // expansion. A user property may intentionally return distinct @@ -573,13 +585,16 @@ private static String resolveUserDefinedProperty(RuntimeScalar codeRef, String s String definition = result.elements.getFirst().toString(); // Parse and cache the property definition - return parseUserDefinedProperty(definition, recursionSet, subName); + return parseUserDefinedProperty(definition, recursionSet, diagnosticName); + } catch (PerlDieException e) { + throw propertyDefinitionDie(e, diagnosticName); } catch (PerlCompilerException e) { // Re-throw Perl exceptions (like die in IsDeath) String msg = e.getMessage(); if (msg != null && !msg.contains("in expansion of")) { - throw new IllegalArgumentException("Died" + (msg.isEmpty() ? "" : ": " + msg) + " in expansion of " + subName, e); + throw new IllegalArgumentException("Died" + (msg.isEmpty() ? "" : ": " + msg) + + " in expansion of " + diagnosticName, e); } throw e; } catch (IllegalArgumentException e) { @@ -591,12 +606,31 @@ private static String resolveUserDefinedProperty(RuntimeScalar codeRef, String s } } + private static IllegalArgumentException propertyDefinitionDie( + PerlDieException failure, String propertyName) { + String message = failure.getMessage(); + return new IllegalArgumentException( + "Error \"" + (message == null ? "" : message) + + "\" in expansion of " + propertyName); + } + /** * Resolve deferred top-level user properties before entering the synchronized * regex compiler. Property subs are arbitrary Perl and may block; keeping * them outside that compiler monitor lets unrelated property names proceed. */ static void preloadUserDefinedProperties(String pattern, boolean caseInsensitive) { + preloadUserDefinedProperties(pattern, caseInsensitive, false); + } + + static void preloadDeferredUserDefinedProperties( + String pattern, boolean caseInsensitive) { + preloadUserDefinedProperties(pattern, caseInsensitive, true); + } + + private static void preloadUserDefinedProperties( + String pattern, boolean caseInsensitive, + boolean qualifyBareDiagnosticName) { if (pattern == null || pattern.isEmpty()) return; for (int slash = pattern.indexOf('\\'); slash >= 0; @@ -622,19 +656,20 @@ static void preloadUserDefinedProperties(String pattern, boolean caseInsensitive // translator here would turn that intentional forward reference // into an early "unsupported property" fatal before the // placeholder path can run. - tryUserDefinedProperty(property, new HashSet<>(), caseInsensitive); + tryUserDefinedProperty(property, new LinkedHashSet<>(), caseInsensitive, + qualifyBareDiagnosticName); } slash = end; } } public static String translateUnicodeProperty(String property, boolean negated) { - return translateUnicodeProperty(property, negated, new HashSet<>(), false); + return translateUnicodeProperty(property, negated, new LinkedHashSet<>(), false); } static String translateUnicodeProperty( String property, boolean negated, boolean caseInsensitive) { - return translateUnicodeProperty(property, negated, new HashSet<>(), caseInsensitive); + return translateUnicodeProperty(property, negated, new LinkedHashSet<>(), caseInsensitive); } private static String translateUnicodeProperty(String property, boolean negated, Set recursionSet) { diff --git a/src/test/resources/unit/regex/user_property_definition_diagnostics.t b/src/test/resources/unit/regex/user_property_definition_diagnostics.t new file mode 100644 index 0000000000..196d283920 --- /dev/null +++ b/src/test/resources/unit/regex/user_property_definition_diagnostics.t @@ -0,0 +1,34 @@ +use strict; +use warnings; +use utf8; +use Test::More tests => 12; + +sub IsOverflow { return "0\t80000000000000000#ネ" } +sub IsRangeReversed { return "200 100#ネ" } +sub IsNonHex { return "BEEF CAGED#ネ" } +sub IsDeath { die "intentional property death\n" } +sub InRecursedA { return "+main::InRecursedB\n" } +sub InRecursedB { return "+main::InRecursedC\n" } +sub InRecursedC { return "+main::InRecursedA\n" } +sub InOneBadApple { return "0100\t0110\n10000\t10010\nF000\\tF010\n0400\t0410" } + +my @cases = ( + [ IsOverflow => qr/^Code point too large in "0\t80000000000000000#ネ" in expansion of main::IsOverflow/ ], + [ IsRangeReversed => qr/^Illegal range in "200 100#ネ" in expansion of main::IsRangeReversed/ ], + [ IsNonHex => qr/^Can't find Unicode property definition "BEEF CAGED" in expansion of main::IsNonHex/ ], + [ IsDeath => qr/^Error "intentional property death\n" in expansion of main::IsDeath/s ], + [ InRecursedA => qr/^Infinite recursion in user-defined property "main::InRecursedA" in expansion of main::InRecursedC in expansion of main::InRecursedB in expansion of main::InRecursedA/ ], +); + +for my $case (@cases) { + my ($property, $diagnostic) = @$case; + my $source = 'qr/\p{main::' . $property . '}/'; + my $ok = eval "$source; 1"; + ok(!$ok, "$property definition is rejected"); + like($@, $diagnostic, "$property reports the Perl expansion diagnostic"); +} + +my $nested_ok = eval q{qr/\p{InOneBadApple}/; 1}; +ok(!$nested_ok, 'a bad nested component rejects the user property'); +like($@, qr/^Can't find Unicode property definition "F000\\tF010" in expansion of InOneBadApple/, + 'a bad nested component reports the unqualified expansion name'); From ec39e3225d696fa326062317aa6c97d98ab34b30 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:10:25 +0200 Subject: [PATCH 09/60] fix(regex): preserve deferred Unicode property provenance (#1013) Squashed integration of PR #1013 at 9eae6b895. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 25 +++++++----- .../runtime/regex/RuntimeRegex.java | 6 +++ .../runtime/regex/UnicodeResolver.java | 10 ++++- .../runtimetypes/RuntimeRegexState.java | 4 ++ .../regex/user_property_source_provenance.t | 39 +++++++++++++++++++ 5 files changed, 74 insertions(+), 10 deletions(-) create mode 100644 src/test/resources/unit/regex/user_property_source_provenance.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 3b1d2767cf..367de673c2 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -233,9 +233,12 @@ user-property-definition diagnostic assertions as separate work. Direct user-property definitions now report Perl-compatible overflow, reversed range, invalid component, callback death, recursion-chain, and package-name diagnostics. The focused 12-assertion oracle passes on both execution backends; -`regexp_unicode_prop.t` gains two more assertions. Four deferred bare-name -assertions retain correct diagnostic content but still need literal source-order -provenance to add the canonical `main::` prefix. +`regexp_unicode_prop.t` gains two more assertions. Deferred property provenance +now survives implicit Unicode-flag regex copies and remains associated with the +property name for later source-literal reuse. The focused 8-assertion oracle +passes on both execution backends; unchanged upstream coverage gains nine more +assertions to reach 1,065/1,110, and all user-property diagnostic assertions +through test 1,094 pass. Executable callback source and literal trailing `/x` comments survive canonical regex-object stringification on both execution backends. Recursive Joni call @@ -359,9 +362,12 @@ matcher-specific timeouts on both execution backends. - [x] Matched user-property definition validation, deterministic recursion chains, callback-death wrapping, and direct package-name policy. The focused oracle passes 12/12 on system Perl, JVM, and interpreter; - unchanged upstream coverage gains two assertions, with four source-order - provenance assertions tracked separately. - - [ ] Complete the remaining Unicode aliases and diagnostic parity inventory. + unchanged upstream coverage gains two assertions. + - [x] Preserved deferred user-property package provenance through implicit + Unicode-flag copies and later literal reuse. The focused oracle passes 8/8 + on system Perl, JVM, and interpreter; `regexp_unicode_prop.t` gains nine + assertions to 1,065/1,110 on both execution backends. + - [ ] Complete the remaining built-in Unicode aliases. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) - [x] Preserved mixed executable-source provenance, nested dynamic callback @@ -386,9 +392,10 @@ matcher-specific timeouts on both execution backends. slices, then rerun the forced-Joni corpus from the new combined head. 3. Capture the clean-branch JVM and interpreter 80-file baselines and compare both to PR 958 with the regression exit gate. -4. Integrate the invalid-property and definition-diagnostic slices, then finish - PerlOnJava4's built-in Unicode aliases and preserve compile-time source-order - provenance for the four deferred bare-name diagnostics. +4. Integrate PR #1011 and the deferred-property provenance slice, then finish + PerlOnJava4's built-in Unicode aliases against the 1,065/1,110 diagnostics + baseline. Fixing the 29 executed alias failures also unlocks 16 gated match + assertions, so the alias-complete target is 1,110/1,110. 5. Keep the warning-free whole-unit-suite gate green with Joni as the default; use explicit Java mode only to classify corpus regressions before Phase 5 removes the legacy backend and selector. diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java index 63c3c9d87d..46b83f9712 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java @@ -1351,6 +1351,8 @@ && containsExecutableSource(patternString.toString(), modifierStr.indexOf('x') > regex.patternNoInternalMarkers = originalRegex.patternNoInternalMarkers; regex.patternUnicodeNoInternalMarkers = originalRegex.patternUnicodeNoInternalMarkers; regex.patternString = originalRegex.patternString; + regex.deferredUserDefinedUnicodeProperties = + originalRegex.deferredUserDefinedUnicodeProperties; regex.hasPreservesMatch = originalRegex.hasPreservesMatch; regex.quoteConstruction = originalRegex.quoteConstruction; regex.warningsOnUse = new ArrayList<>(originalRegex.warningsOnUse); @@ -1391,6 +1393,8 @@ && containsExecutableSource(patternString.toString(), modifierStr.indexOf('x') > regex.patternNoInternalMarkers = originalRegex.patternNoInternalMarkers; regex.patternUnicodeNoInternalMarkers = originalRegex.patternUnicodeNoInternalMarkers; regex.patternString = originalRegex.patternString; + regex.deferredUserDefinedUnicodeProperties = + originalRegex.deferredUserDefinedUnicodeProperties; regex.hasPreservesMatch = originalRegex.hasPreservesMatch; regex.quoteConstruction = originalRegex.quoteConstruction; regex.warningsOnUse = new ArrayList<>(originalRegex.warningsOnUse); @@ -1591,6 +1595,8 @@ public static RuntimeScalar getReplacementRegex(RuntimeScalar patternString, Run regex.patternNoInternalMarkers = resolvedRegex.patternNoInternalMarkers; regex.patternUnicodeNoInternalMarkers = resolvedRegex.patternUnicodeNoInternalMarkers; regex.patternString = resolvedRegex.patternString; + regex.deferredUserDefinedUnicodeProperties = + resolvedRegex.deferredUserDefinedUnicodeProperties; regex.regexFlags = resolvedRegex.regexFlags; regex.hasPreservesMatch = resolvedRegex.hasPreservesMatch; regex.quoteConstruction = resolvedRegex.quoteConstruction; diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index 0942dc00a7..ff71ba3697 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -34,6 +34,10 @@ private static Map userPropertyCache() { return PerlRuntime.current().regexState().userUnicodePropertyCache; } + private static Set deferredUserProperties() { + return PerlRuntime.current().regexState().deferredUserUnicodeProperties; + } + private static String userPropertyCacheKey(String subName, boolean caseInsensitive) { return subName + (caseInsensitive ? "\u0000i" : "\u0000s"); } @@ -537,7 +541,8 @@ private static String tryUserDefinedProperty( RuntimeScalar codeRef = GlobalVariable.getGlobalCodeRef(subName); final String resolvedSubName = subName; - final String diagnosticName = qualifyBareDiagnosticName + final String diagnosticName = (qualifyBareDiagnosticName + || deferredUserProperties().contains(subName)) && !property.contains("::") ? subName : property; final String coordinationKey = cacheKey; try { @@ -649,6 +654,9 @@ private static void preloadUserDefinedProperties( String property = pattern.substring(slash + 3, end).trim(); if (property.startsWith("^")) property = property.substring(1).trim(); if (isUserDefinedPropertyName(property)) { + if (qualifyBareDiagnosticName && !property.contains("::")) { + deferredUserProperties().add("main::" + property); + } // Preloading is an optimization for callbacks that are already // available. Perl permits qr// to contain a forward reference // to a user property; RegexPreprocessor represents that with a diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeRegexState.java b/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeRegexState.java index 5ffe84acff..f84d1d3266 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeRegexState.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeRegexState.java @@ -4,9 +4,11 @@ import org.perlonjava.runtime.regex.RuntimeRegex; import java.util.LinkedHashMap; +import java.util.LinkedHashSet; import java.util.ArrayList; import java.util.List; import java.util.Map; +import java.util.Set; /** * Mutable regular-expression state owned by one {@link PerlRuntime}. @@ -46,6 +48,7 @@ public final class RuntimeRegexState { /** Stable scalar identities for literal regex targets, keyed by compiled call site. */ public final Map literalRegexTargets = new LinkedHashMap<>(); public final Map userUnicodePropertyCache = new LinkedHashMap<>(); + public final Set deferredUserUnicodeProperties = new LinkedHashSet<>(); /** * Per-runtime compiled templates. Some templates support deferred runtime @@ -101,5 +104,6 @@ public void clearMatchState() { /** Copy immutable regex metadata that Perl ithreads inherit at creation. */ void snapshotInto(RuntimeRegexState target) { target.userUnicodePropertyCache.putAll(userUnicodePropertyCache); + target.deferredUserUnicodeProperties.addAll(deferredUserUnicodeProperties); } } diff --git a/src/test/resources/unit/regex/user_property_source_provenance.t b/src/test/resources/unit/regex/user_property_source_provenance.t new file mode 100644 index 0000000000..21ff58c5b9 --- /dev/null +++ b/src/test/resources/unit/regex/user_property_source_provenance.t @@ -0,0 +1,39 @@ +use strict; +use warnings; +use feature 'unicode_strings'; +use Test::More tests => 8; + +our @deferred; +BEGIN { + for my $name (qw(IsP36Overflow IsP36Reverse IsP36NonHex IsP36Death)) { + push @deferred, qr/\p{$name}/; + } +} + +my @expected = ( + qr/Code point too large .* expansion of main::IsP36Overflow/s, + qr/Illegal range .* expansion of main::IsP36Reverse/s, + qr/Can't find Unicode property definition .* expansion of main::IsP36NonHex/s, + qr/P36 callback death.* expansion of main::IsP36Death/s, +); + +my @names = qw(IsP36Overflow IsP36Reverse IsP36NonHex IsP36Death); +for my $index (0 .. $#deferred) { + undef $@; + eval { "A" =~ $deferred[$index] }; + like($@, $expected[$index], + "deferred literal records canonical user-property package provenance"); +} + +for my $index (0 .. $#names) { + undef $@; + my $name = $names[$index]; + eval { "A" =~ /\p{$name}/ }; + like($@, $expected[$index], + "source literal retains canonical user-property package provenance"); +} + +sub IsP36Overflow { "0 80000000000000000\n" } +sub IsP36Reverse { "200 100\n" } +sub IsP36NonHex { "BEEF CAGED\n" } +sub IsP36Death { die "P36 callback death\n" } From 511f67fcd29e514e0c5f9eebdcfc547fb6f21436 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:10:26 +0200 Subject: [PATCH 10/60] fix(regex): execute inline p modifiers through Joni (#1014) Squashed integration of PR #1014 at 785aa06a6. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- .../perlonjava/runtime/regex/RegexFlags.java | 46 ++++++++++++++++++ .../runtime/regex/RuntimeRegex.java | 21 +++++++- .../unit/regex/regex_inline_p_modifier.t | 44 +++++++++++++++++ third_party/joni/src/org/joni/Parser.java | 13 +++-- .../joni/test/TestPerlPreserveModifier.java | 48 +++++++++++++++++++ 5 files changed, 166 insertions(+), 6 deletions(-) create mode 100644 src/test/resources/unit/regex/regex_inline_p_modifier.t create mode 100644 third_party/joni/test/org/joni/test/TestPerlPreserveModifier.java diff --git a/src/main/java/org/perlonjava/runtime/regex/RegexFlags.java b/src/main/java/org/perlonjava/runtime/regex/RegexFlags.java index 85f530276b..6571bd2c81 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RegexFlags.java +++ b/src/main/java/org/perlonjava/runtime/regex/RegexFlags.java @@ -75,6 +75,52 @@ public static void validateModifiers(String modifiers) { } } + /** + * Finds a positive inline Perl {@code p} modifier without treating escaped + * text or character-class contents as modifier groups. Match-variable + * retention is PerlOnJava source policy rather than a Joni matcher option. + */ + static boolean hasInlinePreserveModifier(String pattern) { + if (pattern == null || pattern.length() < 4) return false; + + boolean escaped = false; + boolean inClass = false; + for (int i = 0; i + 3 < pattern.length(); i++) { + char c = pattern.charAt(i); + if (escaped) { + escaped = false; + continue; + } + if (c == '\\') { + escaped = true; + continue; + } + if (c == '[') { + inClass = true; + continue; + } + if (c == ']' && inClass) { + inClass = false; + continue; + } + if (inClass || c != '(' || pattern.charAt(i + 1) != '?') continue; + + boolean negative = false; + for (int j = i + 2; j < pattern.length(); j++) { + char modifier = pattern.charAt(j); + if (modifier == ':' || modifier == ')') break; + if (modifier == '-') { + negative = true; + continue; + } + if (modifier == '^') continue; + if (modifier < 'a' || modifier > 'z') break; + if (modifier == 'p' && !negative) return true; + } + } + return false; + } + public int toPatternFlags() { int flags = 0; diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java index 46b83f9712..c64fce0b9e 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java @@ -653,12 +653,15 @@ private static synchronized RuntimeRegex compileSynchronized( regex.recursivePattern.hasDeferredUserDefinedUnicodeProperty() || regex.recursivePatternUnicode .hasDeferredUserDefinedUnicodeProperty(); - regex.hasPreservesMatch = regex.regexFlags.preservesMatch(); + regex.hasPreservesMatch = regex.regexFlags.preservesMatch() + || RegexFlags.hasInlinePreserveModifier(compilePatternString); regex.hasBranchReset = false; regex.hasBackslashK = false; } else { regex.deferredUserDefinedUnicodeProperties = RegexPreprocessor.hadDeferredUnicodePropertyEncountered(); - regex.hasPreservesMatch = regex.regexFlags.preservesMatch() || RegexPreprocessor.hadInlinePFlag(); + regex.hasPreservesMatch = regex.regexFlags.preservesMatch() + || RegexFlags.hasInlinePreserveModifier(compilePatternString) + || RegexPreprocessor.hadInlinePFlag(); regex.hasBranchReset = RegexPreprocessor.hadBranchReset(); regex.hasBackslashK = RegexPreprocessor.hadBackslashK(); regex.warningsOnUse.addAll(RegexPreprocessor.getWarningsOnUse()); @@ -2053,6 +2056,11 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc int capture = 1; int previousPos = startPos; // Track the previous position int previousMatchEnd = -1; // Track end of previous match + boolean previousMatchUsedPFlag = regexState.lastMatchUsedPFlag; + // Inline /p is Perl match-variable policy rather than a Joni option. + // Publish it while callouts execute; restore the preceding successful + // match's policy if this matcher ultimately fails. + regexState.lastMatchUsedPFlag = regex.hasPreservesMatch; // NOTE: Do NOT clear global match variables here. // // Perl preserves $1, @-, @+, $&, etc. from the last *successful* match even if a @@ -2184,6 +2192,10 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc found = false; } + if (!found) { + regexState.lastMatchUsedPFlag = previousMatchUsedPFlag; + } + // Reset pos() on failed match with /g, unless /c is set if (!found && regex.regexFlags.isGlobalMatch() && !regex.regexFlags.keepCurrentPosition() && posScalar != null) { posScalar.set(scalarUndef); @@ -2799,6 +2811,8 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar // The result string after substitutions StringBuilder resultBuffer = new StringBuilder(); int found = 0; + boolean previousMatchUsedPFlag = state().lastMatchUsedPFlag; + state().lastMatchUsedPFlag = regex.hasPreservesMatch; // Unwrap readonly scalar if (replacement.type == RuntimeScalarType.READONLY_SCALAR) replacement = (RuntimeScalar) replacement.value; @@ -2964,6 +2978,9 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar WarnDie.warn(new RuntimeScalar(e.getMessage() + "\n"), RuntimeScalarCache.scalarEmptyString); found = 0; } + if (found == 0) { + state().lastMatchUsedPFlag = previousMatchUsedPFlag; + } // Append the remaining text after the last match to the result buffer resultBuffer.append(inputStr, lastAppendEnd, inputStr.length()); diff --git a/src/test/resources/unit/regex/regex_inline_p_modifier.t b/src/test/resources/unit/regex/regex_inline_p_modifier.t new file mode 100644 index 0000000000..cad265b7e9 --- /dev/null +++ b/src/test/resources/unit/regex/regex_inline_p_modifier.t @@ -0,0 +1,44 @@ +use strict; +use warnings; +use Test::More tests => 15; + +{ + my $subject = '012-345-6789'; + ok($subject =~ /(?p)345/, 'standalone inline p matches'); + is(${^PREMATCH}, '012-', 'standalone inline p exposes prematch'); + is(${^MATCH}, '345', 'standalone inline p exposes match'); + is(${^POSTMATCH}, '-6789', 'standalone inline p exposes postmatch'); +} + +{ + my $subject = '012-345-6789'; + ok($subject =~ /(?p:345)/, 'scoped inline p matches'); + is(${^MATCH}, '345', 'scoped inline p exposes match'); +} + +{ + my $subject = '012-AbC-6789'; + ok($subject =~ /(?ip:abc)/, 'inline p composes with matcher modifiers'); + is(${^MATCH}, 'AbC', 'combined inline modifiers expose exact match'); +} + +{ + my $literal = '(?p)'; + ok($literal =~ /\(\?p\)/, 'escaped modifier spelling remains literal'); + ok(!defined ${^MATCH}, 'escaped modifier spelling does not enable p'); +} + +{ + my $literal = 'p'; + ok($literal =~ /[(?p)]/, 'modifier spelling in a class remains literal'); + ok(!defined ${^MATCH}, 'character-class spelling does not enable p'); +} + +{ + my $callback_match; + my $subject = 'a'; + is($subject =~ s/(?p:a(?{ $callback_match = ${^MATCH} }))/b/, 1, + 'inline p substitution callback executes'); + is($callback_match, 'a', 'inline p is visible to substitution callback'); + is($subject, 'b', 'inline p substitution retains normal replacement behavior'); +} diff --git a/third_party/joni/src/org/joni/Parser.java b/third_party/joni/src/org/joni/Parser.java index 5197662710..68177885c4 100644 --- a/third_party/joni/src/org/joni/Parser.java +++ b/third_party/joni/src/org/joni/Parser.java @@ -629,13 +629,13 @@ private Node parseEnclose(TokenType term) { newSyntaxException(UNDEFINED_GROUP_OPTION); } - // case 'p': #ifdef USE_POSIXLINE_OPTION case '-': case 'i': case 'm': case 's': case 'x': case 'n': + case 'p': case 'a': case 'd': case 'l': @@ -680,9 +680,14 @@ private Node parseEnclose(TokenType term) { newSyntaxException(UNDEFINED_GROUP_OPTION); } break; - // case 'p': #ifdef USE_POSIXLINE_OPTION // not defined - // option = bsOnOff(option, Option.MULTILINE|Option.SINGLELINE, neg); - // break; + case 'p': + // Perl's /p controls ${^PREMATCH}, ${^MATCH}, and + // ${^POSTMATCH} retention in the host runtime. It has + // no matcher option bit, but is valid inline syntax. + if (!syntax.op2OptionPerl()) { + newSyntaxException(UNDEFINED_GROUP_OPTION); + } + break; case 'a': /* limits \d, \s, \w and POSIX brackets to ASCII range */ if ((syntax.op2OptionPerl() || syntax.op2OptionRuby()) && !neg) { diff --git a/third_party/joni/test/org/joni/test/TestPerlPreserveModifier.java b/third_party/joni/test/org/joni/test/TestPerlPreserveModifier.java new file mode 100644 index 0000000000..e90b00b370 --- /dev/null +++ b/third_party/joni/test/org/joni/test/TestPerlPreserveModifier.java @@ -0,0 +1,48 @@ +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni.test; + +import static org.junit.Assert.assertEquals; + +import java.nio.charset.StandardCharsets; + +import org.jcodings.specific.UTF8Encoding; +import org.joni.Option; +import org.joni.Regex; +import org.joni.Syntax; +import org.junit.Test; + +public class TestPerlPreserveModifier { + private static int search(String pattern, String input) { + byte[] patternBytes = pattern.getBytes(StandardCharsets.UTF_8); + byte[] inputBytes = input.getBytes(StandardCharsets.UTF_8); + Regex regex = new Regex(patternBytes, 0, patternBytes.length, + Option.NONE, UTF8Encoding.INSTANCE, Syntax.PerlNG); + return regex.matcher(inputBytes).search(0, inputBytes.length, Option.NONE); + } + + @Test + public void acceptsPerlPreserveModifierAsMatcherNoOp() { + assertEquals(0, search("(?p)abc", "abc")); + assertEquals(0, search("(?p:abc)", "abc")); + assertEquals(0, search("(?ip:abc)", "ABC")); + assertEquals(0, search("(?-p:abc)", "abc")); + } +} From f402cd52862f13998db99b564614ca7ac2690e3e Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:10:26 +0200 Subject: [PATCH 11/60] fix(regex): preserve regex control escape operands (#1015) Squashed integration of PR #1015 at d5a7b00ed. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 33 ++++++++++++++++--- .../frontend/parser/StringDoubleQuoted.java | 5 +++ .../regex/regex_control_character_range.t | 11 +++++++ 3 files changed, 44 insertions(+), 5 deletions(-) create mode 100644 src/test/resources/unit/regex/regex_control_character_range.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 367de673c2..3edf1180f6 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -240,6 +240,16 @@ passes on both execution backends; unchanged upstream coverage gains nine more assertions to reach 1,065/1,110, and all user-property diagnostic assertions through test 1,094 pass. +Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` +syntax as matcher-neutral policy. PerlOnJava publishes that policy while +ordinary and substitution callbacks execute, without misclassifying escaped or +character-class text. The focused 15-assertion oracle passes on system Perl, +JVM, and interpreter, and unchanged `reg_pmod.t` reaches 88/88 on both +execution backends. Regex source scanning also consumes each `\c` operand +before interpolation, so `\c@` cannot be mistaken for `@-`; the focused +4-assertion oracle passes on all three runtimes and unchanged `subst.t` reaches +250/281 on JVM and interpreter. + Executable callback source and literal trailing `/x` comments survive canonical regex-object stringification on both execution backends. Recursive Joni call frames now preserve the Perl-visible caller capture view for optimistic @@ -367,6 +377,13 @@ matcher-specific timeouts on both execution backends. Unicode-flag copies and later literal reuse. The focused oracle passes 8/8 on system Perl, JVM, and interpreter; `regexp_unicode_prop.t` gains nine assertions to 1,065/1,110 on both execution backends. + - [x] Accepted inline `p` directly in Joni while retaining match-variable + policy in PerlOnJava, including provisional callback state. The focused + oracle passes 15/15 and unchanged `reg_pmod.t` passes 88/88 on JVM and + interpreter. + - [x] Preserved `\c` control operands through regex source interpolation. + The focused oracle passes 4/4 and unchanged `subst.t` gains test 154 on + both execution backends. - [ ] Complete the remaining built-in Unicode aliases. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) @@ -392,11 +409,17 @@ matcher-specific timeouts on both execution backends. slices, then rerun the forced-Joni corpus from the new combined head. 3. Capture the clean-branch JVM and interpreter 80-file baselines and compare both to PR 958 with the regression exit gate. -4. Integrate PR #1011 and the deferred-property provenance slice, then finish - PerlOnJava4's built-in Unicode aliases against the 1,065/1,110 diagnostics - baseline. Fixing the 29 executed alias failures also unlocks 16 gated match - assertions, so the alias-complete target is 1,110/1,110. -5. Keep the warning-free whole-unit-suite gate green with Joni as the default; +4. Integrate PR #1011, the deferred-property provenance slice, inline-`p` + syntax, and control-escape source handling. Finish PerlOnJava4's built-in + Unicode aliases against the 1,065/1,110 diagnostics baseline. Fixing the 29 + executed alias failures also unlocks 16 gated match assertions, so the + alias-complete target is 1,110/1,110. +5. Give the matcher adapter an independent Joni `gpos` cursor. Substitution + must search before `pos` when pattern atoms precede `\G`; the current + start-at-`pos` shortcut causes `subst.t` tests 165-188 to miss every match. + Keep Java's temporary approximation isolated and use Joni's existing + `Matcher.search(gpos, start, range, option)` API for final semantics. +6. Keep the warning-free whole-unit-suite gate green with Joni as the default; use explicit Java mode only to classify corpus regressions before Phase 5 removes the legacy backend and selector. diff --git a/src/main/java/org/perlonjava/frontend/parser/StringDoubleQuoted.java b/src/main/java/org/perlonjava/frontend/parser/StringDoubleQuoted.java index 45438a02b8..fde477ec50 100644 --- a/src/main/java/org/perlonjava/frontend/parser/StringDoubleQuoted.java +++ b/src/main/java/org/perlonjava/frontend/parser/StringDoubleQuoted.java @@ -499,6 +499,11 @@ private void parseDoubleQuotedEscapesRegex() { caseModifiers.push(new CaseModifier("Q", false)); } + // A regex control escape consumes its operand before normal string + // interpolation sees it. In particular, the '@' in \c@ is not the + // start of @- when used in a range such as [\c@-\c_]. + case "c" -> appendToCurrentSegment("\\c" + TokenUtils.consumeChar(parser)); + // Unknown escape - treat as literal character default -> appendToCurrentSegment("\\" + escape); } diff --git a/src/test/resources/unit/regex/regex_control_character_range.t b/src/test/resources/unit/regex/regex_control_character_range.t new file mode 100644 index 0000000000..c2eaa1b71f --- /dev/null +++ b/src/test/resources/unit/regex/regex_control_character_range.t @@ -0,0 +1,11 @@ +use strict; +use warnings; +use Test::More tests => 4; + +ok("\x00" =~ /[\c@-\c_]/, 'NUL is in the control-character range'); +ok("\x1f" =~ /[\c@-\c_]/, 'unit separator is in the control-character range'); +ok("\x20" !~ /[\c@-\c_]/, 'space is outside the control-character range'); + +my $controls = "\x20\x00\x30\x01\x40\x1a\x50\x1f\x60"; +is($controls =~ s/[\c@-\c_]//gr, "\x20\x30\x40\x50\x60", + 'substitution removes the complete control-character range'); From 884c6b57aefb4c5c6c7f18d420949fdf9443cfec Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:11:32 +0200 Subject: [PATCH 12/60] fix(regex): integrate Joni Unicode and G-position parity (#1016) Squashed integration of PR #1016 at b43687d3f. PRs #1008, #1009, and #1012 are preserved as preceding integration commits; temporary join merges are intentionally excluded. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 47 ++++++----- .../runtime/regex/JoniRegexPattern.java | 22 ++++- .../runtime/regex/RegexMatcher.java | 6 ++ .../runtime/regex/RuntimeRegex.java | 22 ++++- .../runtime/regex/UnicodeResolver.java | 82 +++++++++++++++++++ .../regex/regex_gpos_independent_search.t | 35 ++++++++ .../unit/regex/unicode_property_aliases.t | 20 +++++ 7 files changed, 205 insertions(+), 29 deletions(-) create mode 100644 src/test/resources/unit/regex/regex_gpos_independent_search.t create mode 100644 src/test/resources/unit/regex/unicode_property_aliases.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 3edf1180f6..acab2e8249 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -250,6 +250,12 @@ before interpolation, so `\c@` cannot be mistaken for `@-`; the focused 4-assertion oracle passes on all three runtimes and unchanged `subst.t` reaches 250/281 on JVM and interpreter. +The matcher adapter now carries Joni's search start and Perl `\G` position as +independent cursors, including Unicode offset conversion. The focused +12-assertion oracle passes on system Perl, JVM, and interpreter; unchanged +`subst.t` reaches 275/281 on both execution backends with tests 165-188 +restored. The temporary Java backend retains its start-at-`pos` approximation. + Executable callback source and literal trailing `/x` comments survive canonical regex-object stringification on both execution backends. Recursive Joni call frames now preserve the Perl-visible caller capture view for optimistic @@ -322,6 +328,10 @@ matcher-specific timeouts on both execution backends. `/g` hot loop. The focused million-match oracle completes in 1.07 seconds on JVM and 1.41 seconds on interpreter (PR #1008), with exact map and supplementary-character capture-boundary coverage. + - [x] Separated Joni's search-start and `\G` cursors for ordinary matching + and substitution, including Unicode subjects and code replacements. The + focused oracle passes 12/12 and unchanged `subst.t` passes 275/281 on JVM + and interpreter. - [x] Phase 2: Conditions and backtracking-visible state (2026-08-17) - [x] Implemented executable callback conditions, control verbs including `(*MARK:NAME)`, and callback-visible recursive capture state in Joni. @@ -358,7 +368,7 @@ matcher-specific timeouts on both execution backends. system Perl, JVM, and interpreter. - [x] Removed the final `pat.t.patch` hunk, deleted the patch and its importer configuration, and resynchronized the unmodified Perl 5.44 `pat.t`. -- [ ] Phase 3: Unicode and pattern syntax completion (in progress) +- [x] Phase 3: Unicode and pattern syntax completion (2026-08-17) - [x] Added Perl escape syntax, Unicode-property resolution, scoped ASCII folds, possessive intervals, and bounded lookbehind support to Joni. - [x] Converted public regex `pos` values between Perl logical-character @@ -384,7 +394,10 @@ matcher-specific timeouts on both execution backends. - [x] Preserved `\c` control operands through regex source interpolation. The focused oracle passes 4/4 and unchanged `subst.t` gains test 154 on both execution backends. - - [ ] Complete the remaining built-in Unicode aliases. + - [x] Completed Perl-compatible built-in Unicode alias normalization while + preserving deferred user-property precedence. The focused alias oracle + passes 16/16 on system Perl, JVM, and interpreter; unchanged + `regexp_unicode_prop.t` passes 1,110/1,110 on both execution backends. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) - [x] Preserved mixed executable-source provenance, nested dynamic callback @@ -400,26 +413,16 @@ matcher-specific timeouts on both execution backends. ### Next Steps -1. Remediate the remaining forced-Joni failure causes, starting with the - catastrophic `regexp*` cluster and absent generated Unicode fixtures; keep - per-child hard limits throughout. PR #1008 has removed the release-blocking - `pat_psycho*` offset-map reconstruction and bounded profiling confirms that - matcher-wrapper reuse is no longer on the critical path. -2. Integrate the stacked Joni offset-map, regex-set property, and CEC timeout - slices, then rerun the forced-Joni corpus from the new combined head. -3. Capture the clean-branch JVM and interpreter 80-file baselines and compare - both to PR 958 with the regression exit gate. -4. Integrate PR #1011, the deferred-property provenance slice, inline-`p` - syntax, and control-escape source handling. Finish PerlOnJava4's built-in - Unicode aliases against the 1,065/1,110 diagnostics baseline. Fixing the 29 - executed alias failures also unlocks 16 gated match assertions, so the - alias-complete target is 1,110/1,110. -5. Give the matcher adapter an independent Joni `gpos` cursor. Substitution - must search before `pos` when pattern atoms precede `\G`; the current - start-at-`pos` shortcut causes `subst.t` tests 165-188 to miss every match. - Keep Java's temporary approximation isolated and use Joni's existing - `Matcher.search(gpos, start, range, option)` API for final semantics. -6. Keep the warning-free whole-unit-suite gate green with Joni as the default; +1. Complete the combined offset-map, regex-set, Joni safeguard, property + diagnostics/provenance, Unicode-alias, inline-`p`, and control-escape + integration while retaining each validated prerequisite head. +2. Integrate the lossless generated Unicode fixtures, then rerun the + forced-Joni corpus from the combined head with every child hard-bounded. +3. Capture clean JVM and interpreter 80-file baselines and compare every file + with PR 958 under the no-regression gate. +4. Classify residual engine-introspection exclusions, then queue the largest + remaining semantic cluster as a focused slice. +5. Keep the warning-free whole-unit-suite gate green with Joni as the default; use explicit Java mode only to classify corpus regressions before Phase 5 removes the legacy backend and selector. diff --git a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java index c7d587428d..5028248612 100644 --- a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java +++ b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java @@ -165,12 +165,13 @@ private static UserPropertyTranslation translateUserDefinedProperties( "(?i)^(?:scx|script[_ ]?extensions)\\s*=.*"); boolean frontendProperty = unnegated.matches( "(?i)^(?:script|block|blk|age|in|present[_ ]?in)\\s*=.*"); + boolean perlBuiltInAlias = UnicodeResolver.isPerlBuiltInPropertyAlias(unnegated); if (frontendProperty && extendedClassBracketDepth > 0) { translated.append(pattern, i, end + 1); i = end; continue; } - if (!userDefined && !scriptExtensions && !frontendProperty) { + if (!userDefined && !scriptExtensions && !frontendProperty && !perlBuiltInAlias) { translated.append(pattern, i, end + 1); i = end; continue; @@ -629,6 +630,7 @@ private static final class JoniRegexMatcher implements RegexMatcher { private int regionStart; private int regionEnd; private int nextStart; + private int globalPosition = -1; private boolean matched; private int committedLastClosedCapture = -1; private final boolean hasControlVerbState; @@ -679,9 +681,15 @@ private boolean find(int option, boolean anchored) { } int result; try { - result = anchored - ? matcher.match(charToByte[nextStart], charToByte[regionEnd], option) - : matcher.search(charToByte[nextStart], charToByte[regionEnd], option); + if (globalPosition >= 0) { + result = matcher.search(charToByte[globalPosition], charToByte[nextStart], + charToByte[regionEnd], option); + if (anchored && result != charToByte[nextStart]) result = -1; + } else { + result = anchored + ? matcher.match(charToByte[nextStart], charToByte[regionEnd], option) + : matcher.search(charToByte[nextStart], charToByte[regionEnd], option); + } } catch (RuntimeException | Error failure) { if (calloutHandler != null) calloutHandler.abort(); throw failure; @@ -723,6 +731,12 @@ public void region(int start, int end) { @Override public void useAnchoringBounds(boolean enabled) { } @Override public void useTransparentBounds(boolean enabled) { } + @Override + public boolean setGlobalPosition(int position) { + if (position < 0 || position > input.length()) return false; + globalPosition = position; + return true; + } @Override public int start() { return toCharOffset(matcher.getBegin()); } @Override public int end() { return toCharOffset(matcher.getEnd()); } @Override public int start(int index) { return groupOffset(index, true); } diff --git a/src/main/java/org/perlonjava/runtime/regex/RegexMatcher.java b/src/main/java/org/perlonjava/runtime/regex/RegexMatcher.java index 827c5ab5f5..d1c3a19327 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RegexMatcher.java +++ b/src/main/java/org/perlonjava/runtime/regex/RegexMatcher.java @@ -19,6 +19,12 @@ default boolean findNotEmpty() { void useTransparentBounds(boolean enabled); + /** + * Set an independent Perl {@code \G} anchor while retaining the current + * search start. Backends without a separate gpos cursor return false. + */ + default boolean setGlobalPosition(int position) { return false; } + int start(); int end(); diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java index c64fce0b9e..94733dcc18 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java @@ -1937,6 +1937,7 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc RuntimeScalar posScalar = null; boolean isPosDefined = false; int startPos = 0; + boolean nativeGlobalPosition = false; // Flag to skip the first find() when the notempty variant already found a match boolean skipFirstFind = false; @@ -2017,6 +2018,10 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc } } + if (regex.useGAssertion) { + nativeGlobalPosition = matcher.setGlobalPosition(startPos); + } + if (regex.requiredLiteral != null && !inputStr.contains(regex.requiredLiteral)) { if (DEBUG_REGEX) { System.err.println(" required literal prefilter failed: " + regex.requiredLiteral); @@ -2040,7 +2045,7 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc // Start matching from the current position if defined // (skip if notempty variant already found a match - region() would reset the matcher) if (isPosDefined && !skipFirstFind) { - matcher.region(startPos, inputStr.length()); + matcher.region(nativeGlobalPosition ? 0 : startPos, inputStr.length()); // Disable anchoring bounds so ^ and $ in /m mode anchor only at real // line breaks in the input, not at the artificial region boundary. // Java's default useAnchoringBounds(true) would let ^ match at startPos @@ -2074,7 +2079,8 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc skipFirstFind = false; // If \G is used, ensure the match starts at the expected position. // When pos() is undefined, \G anchors at 0 (the default startPos). - if (regex.useGAssertion && matcher.start() != startPos) { + if (regex.useGAssertion && !nativeGlobalPosition + && matcher.start() != startPos) { break; } @@ -2159,6 +2165,7 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc break; // Break out of the loop after the first match in SCALAR context } else { startPos = matchEnd; + if (nativeGlobalPosition) matcher.setGlobalPosition(startPos); if (posScalar != null) { posScalar.set(RuntimePosLvalue.fromMatcherOffset( string, inputStr, startPos)); @@ -2790,6 +2797,8 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar ? compileNonEmptySubstitutionPattern(pattern) : null; int searchStart = 0; + int globalPosition = 0; + boolean nativeGlobalPosition = false; // Honor pos() when \G is used. `s/\G.../.../` should anchor at // pos($string) so a substitution inserted right after a previous /g @@ -2803,9 +2812,11 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar int startPos = RuntimePosLvalue.toMatcherOffset( string, inputStr, posScalar.getInt()); if (startPos >= 0 && startPos <= inputStr.length()) { - searchStart = startPos; + globalPosition = startPos; } } + nativeGlobalPosition = matcher.setGlobalPosition(globalPosition); + if (!nativeGlobalPosition) searchStart = globalPosition; } // The result string after substitutions @@ -2837,6 +2848,7 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar try { while (searchStart <= inputStr.length()) { setSubstitutionRegion(matcher, searchStart, inputStr.length(), true); + if (nativeGlobalPosition) matcher.setGlobalPosition(globalPosition); if (!matcher.find()) { break; } @@ -2902,6 +2914,7 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar if (matcher.end() > matcher.start()) { searchStart = matcher.end(); + globalPosition = searchStart; continue; } @@ -2917,6 +2930,7 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar // The synthetic (?<=[\s\S]) suffix relies on opaque bounds // so a zero-length match at the region start is rejected. setSubstitutionRegion(retryMatcher, zeroLengthOffset, inputStr.length(), false); + if (nativeGlobalPosition) retryMatcher.setGlobalPosition(zeroLengthOffset); boolean retryFound = nonEmptySubstitutionPattern != null ? retryMatcher.find() : retryMatcher.findNotEmpty(); if (retryFound @@ -2966,12 +2980,14 @@ public static RuntimeBase replaceRegex(RuntimeScalar quotedRegex, RuntimeScalar } } searchStart = retryMatcher.end(); + globalPosition = searchStart; consumedNonEmptyRetry = true; } } if (!consumedNonEmptyRetry) { searchStart = bumpGlobalMatchPosition(inputStr, zeroLengthOffset); + globalPosition = searchStart; } } } catch (RegexTimeoutException e) { diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index ff71ba3697..f1b1142a34 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -360,6 +360,9 @@ private static UnicodeSet resolvePropertyReferenceAsSet(String propRef, Set recursionSet) { + UnicodeSet perlBuiltInAlias = resolvePerlBuiltInPropertyAlias(property); + if (perlBuiltInAlias != null) return perlBuiltInAlias; + // Handle well-known Perl property aliases switch (property) { case "XPosixSpace": case "XPerlSpace": case "SpacePerl": @@ -714,6 +717,11 @@ private static String translateUnicodeProperty(String property, boolean negated, // Property not found - fall through to throw error below } + UnicodeSet perlBuiltInAlias = resolvePerlBuiltInPropertyAlias(property); + if (perlBuiltInAlias != null) { + return wrapCharClass(unicodeSetToJavaPattern(perlBuiltInAlias), negated); + } + // Special cases - Perl XPosix properties not natively supported in Java switch (property) { case "lb=cr": @@ -924,6 +932,80 @@ static boolean isUserDefinedPropertyName(String property) { return property != null && USER_DEFINED_PROPERTY_NAME.matcher(property).matches(); } + static boolean isPerlBuiltInPropertyAlias(String property) { + return resolvePerlBuiltInPropertyAlias(property) != null; + } + + private static UnicodeSet resolvePerlBuiltInPropertyAlias(String property) { + if (property == null) return null; + + String alias = property.trim(); + if (alias.equalsIgnoreCase("L&")) { + UnicodeSet casedLetters = unicodePropertyValueSet( + UProperty.GENERAL_CATEGORY, "UppercaseLetter"); + casedLetters.addAll(unicodePropertyValueSet( + UProperty.GENERAL_CATEGORY, "LowercaseLetter")); + casedLetters.addAll(unicodePropertyValueSet( + UProperty.GENERAL_CATEGORY, "TitlecaseLetter")); + return casedLetters; + } + + int equals = alias.indexOf('='); + if (equals > 0 && loosePropertyName(alias.substring(0, equals)).equals("category")) { + return unicodePropertyValueSet( + UProperty.GENERAL_CATEGORY, alias.substring(equals + 1)); + } + + String blockAlias = alias; + if (alias.length() > 2 && alias.regionMatches(true, 0, "in", 0, 2)) { + int valueStart = 2; + while (valueStart < alias.length()) { + char separator = alias.charAt(valueStart); + if (!Character.isWhitespace(separator) && separator != '-' && separator != '_') break; + valueStart++; + } + if (valueStart >= alias.length()) return null; + blockAlias = alias.substring(valueStart); + } else if (equals >= 0 || unicodePropertyValue(UProperty.SCRIPT, alias) >= 0) { + return null; + } + + int blockValue = unicodePropertyValue(UProperty.BLOCK, blockAlias); + if (blockValue < 0) return null; + String canonicalBlock = UCharacter.getPropertyValueName( + UProperty.BLOCK, blockValue, UProperty.NameChoice.LONG); + if (canonicalBlock == null || !loosePropertyName(blockAlias) + .equals(loosePropertyName(canonicalBlock))) { + return null; + } + return new UnicodeSet().applyIntPropertyValue(UProperty.BLOCK, blockValue); + } + + private static String loosePropertyName(String value) { + StringBuilder normalized = new StringBuilder(value.length()); + for (int i = 0; i < value.length(); i++) { + char ch = value.charAt(i); + if (!Character.isWhitespace(ch) && ch != '-' && ch != '_') { + normalized.append(Character.toLowerCase(ch)); + } + } + return normalized.toString(); + } + + private static UnicodeSet unicodePropertyValueSet(int property, String alias) { + int value = unicodePropertyValue(property, alias); + if (value < 0) return null; + return new UnicodeSet().applyIntPropertyValue(property, value); + } + + private static int unicodePropertyValue(int property, String alias) { + try { + return UCharacter.getPropertyValueEnum(property, alias); + } catch (IllegalArgumentException ignored) { + return -1; + } + } + private static String translatePerlAgeProperty(String property, boolean negated) { int equals = property.indexOf('='); if (equals <= 0 || equals == property.length() - 1) return null; diff --git a/src/test/resources/unit/regex/regex_gpos_independent_search.t b/src/test/resources/unit/regex/regex_gpos_independent_search.t new file mode 100644 index 0000000000..721943af96 --- /dev/null +++ b/src/test/resources/unit/regex/regex_gpos_independent_search.t @@ -0,0 +1,35 @@ +use strict; +use warnings; +use Test::More tests => 12; + +my $subject = '123456'; +pos($subject) = 4; +is($subject =~ s/\d\d\G/7/g, 1, 'atoms before G match once'); +is($subject, '12756', 'short replacement after atoms before G'); + +$subject = '123456'; +pos($subject) = 4; +is($subject =~ s/\d\d\G/789/g, 1, 'long replacement matches once'); +is($subject, '1278956', 'long replacement does not move the input anchor'); + +$subject = '123456'; +pos($subject) = 4; +is($subject =~ s/\d\d(?=\d\G)/7/g, 1, 'lookahead observes independent G'); +is($subject, '17456', 'lookahead replacement uses the earlier match start'); + +$subject = '123456'; +pos($subject) = 4; +my $replacement = sub { '78' }; +is($subject =~ s/\d\d\G/$replacement->()/eg, 1, + 'code replacement preserves independent G'); +is($subject, '127856', 'code replacement result is applied once'); + +$subject = '123456'; +pos($subject) = 4; +ok($subject =~ /\d\d\G/g, 'ordinary global match can begin before G'); +is($&, '34', 'ordinary match ends at the external position'); + +$subject = "\x{3b1}12"; +pos($subject) = 2; +ok($subject =~ /\d\G/g, 'G uses Perl character offsets on Unicode input'); +is($&, '1', 'Unicode match begins before the independent G position'); diff --git a/src/test/resources/unit/regex/unicode_property_aliases.t b/src/test/resources/unit/regex/unicode_property_aliases.t new file mode 100644 index 0000000000..393af2b0c3 --- /dev/null +++ b/src/test/resources/unit/regex/unicode_property_aliases.t @@ -0,0 +1,20 @@ +use strict; +use warnings; +use Test::More tests => 16; + +ok('A' =~ /^\p{L&}$/, 'L& matches a cased letter'); +ok('1' !~ /^\p{L&}$/, 'L& rejects a digit'); +ok(chr(0xAC00) =~ /^\p{HangulSyllables}$/, 'bare Hangul block alias matches'); +ok('A' !~ /^\p{HangulSyllables}$/, 'bare Hangul block alias rejects Latin'); +ok('A' =~ /^\p{Category=UppercaseLetter}$/, 'Category long value matches'); +ok('a' !~ /^\p{Category=UppercaseLetter}$/, 'Category long value rejects lowercase'); +ok(chr(0xFF) =~ /^\p{latin-1 supplement}$/, 'loose bare block alias matches'); +ok(chr(0x100) !~ /^\p{latin-1 supplement}$/, 'loose bare block alias rejects next block'); +ok('A' =~ /^\p{InBasicLatin}$/, 'InBasicLatin matches its block'); +ok(chr(0x80) !~ /^\p{InBasicLatin}$/, 'InBasicLatin rejects next block'); +ok(chr(0x80) =~ /^\p{InLatin1Supplement}$/, 'InLatin1Supplement matches'); +ok('A' !~ /^\p{InLatin1Supplement}$/, 'InLatin1Supplement rejects Basic Latin'); +ok(chr(0x100) =~ /^\p{InLatinExtendedA}$/, 'InLatinExtendedA matches'); +ok(chr(0xFF) !~ /^\p{InLatinExtendedA}$/, 'InLatinExtendedA rejects prior block'); +ok(chr(0x30A1) =~ /^\p{InKatakana}$/, 'InKatakana matches'); +ok(chr(0x3041) !~ /^\p{InKatakana}$/, 'InKatakana rejects Hiragana'); From a63e504d2f295b8916c3b100b04008f5a892a549 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:13:43 +0200 Subject: [PATCH 13/60] test(regex): expose lossless Unicode property corpus (#1018) Squashed integration of PR #1018 at 1d6972a51. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 94 ++++--- dev/import-perl5/README.md | 29 ++ dev/import-perl5/config.yaml | 12 +- dev/import-perl5/sync.pl | 262 +++++++++++++++++- .../t/generated_unicode_testprop.t | 203 ++++++++++++++ docs/reference/feature-matrix.md | 2 +- 6 files changed, 557 insertions(+), 45 deletions(-) create mode 100644 dev/import-perl5/t/generated_unicode_testprop.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index acab2e8249..ae2ab0be00 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -222,23 +222,25 @@ compatibility contract. ## Progress Tracking -### Current Status: Joni default; Phase 1 remediation and Phase 3 Unicode active - -Invalid Unicode properties now remain fatal under -`JPERL_UNIMPLEMENTED=warn`, and only exact `Is`/`In` final-name prefixes enter -Perl's user-property callback path. A 39-assertion standard-Perl oracle passes -on JVM and interpreter; the unchanged `regexp_unicode_prop.t` runner improves -from 1,039/1,110 to 1,054/1,110, leaving built-in aliases and six -user-property-definition diagnostic assertions as separate work. -Direct user-property definitions now report Perl-compatible overflow, reversed -range, invalid component, callback death, recursion-chain, and package-name -diagnostics. The focused 12-assertion oracle passes on both execution backends; -`regexp_unicode_prop.t` gains two more assertions. Deferred property provenance -now survives implicit Unicode-flag regex copies and remains associated with the -property name for later source-literal reuse. The focused 8-assertion oracle -passes on both execution backends; unchanged upstream coverage gains nine more -assertions to reach 1,065/1,110, and all user-property diagnostic assertions -through test 1,094 pass. +### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active + +The combined integration stack now includes immutable Joni offset-map reuse, +regex-set property preservation, bounded catastrophic-backtracking safeguards, +user-property diagnostics and provenance, and focused built-in Unicode alias +normalization. The focused property gates pass 39/39, 12/12, 8/8, and 16/16; +unchanged `regexp_unicode_prop.t` passes 1,110/1,110 on JVM and interpreter. +Draft PR #1016 preserves each validated prerequisite head and passed a +warning-free full `make` on its corrected integration history. + +The lossless generated `TestProp.pl` fixture now exposes ten previously gated +`uniprops*.t` files. The exact-history JVM run executes all 290,912 planned +assertions with zero timeouts or incomplete files, at 175,768 passing and +115,144 failing. PR 958 recorded zero plan for all ten files, so this is new +coverage rather than a regression, but it reopens Phase 3's full-corpus exit +gate. Chunks 06–10 are complete except for 791 failures in chunk 05; chunks +01–04 require semantic clustering before Phase 3 can close. +The interpreter produces the exact same 175,768/290,912 split, with identical +per-file plans and no timeout or incomplete file. Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while @@ -282,8 +284,8 @@ one-level failed callback state. The focused `pat_re_eval.t` gate executes all 555 assertions with 550 semantic assertions passing on both execution backends; the remaining five inspect Perl's optimizer/debug transcript. -The forced-backend differential is running in the parallel PerlOnJava3 -checkout. Its completed forced-Java/JVM leg covers all 80 files at +The last completed forced-backend differential's forced-Java/JVM leg covers all +80 files at 49,923/94,823 versus PR 958's 50,273/94,771. The apparent aggregate regression is dominated by `pat{,_thr}.t` aborting after test 239 on a runtime eval-group policy error; that source-policy slice is assigned independently. The completed @@ -368,7 +370,8 @@ matcher-specific timeouts on both execution backends. system Perl, JVM, and interpreter. - [x] Removed the final `pat.t.patch` hunk, deleted the patch and its importer configuration, and resynchronized the unmodified Perl 5.44 `pat.t`. -- [x] Phase 3: Unicode and pattern syntax completion (2026-08-17) +- [ ] Phase 3: Unicode and pattern syntax completion (focused gates complete; + generated full-corpus remediation active) - [x] Added Perl escape syntax, Unicode-property resolution, scoped ASCII folds, possessive intervals, and bounded lookbehind support to Joni. - [x] Converted public regex `pos` values between Perl logical-character @@ -394,10 +397,18 @@ matcher-specific timeouts on both execution backends. - [x] Preserved `\c` control operands through regex source interpolation. The focused oracle passes 4/4 and unchanged `subst.t` gains test 154 on both execution backends. - - [x] Completed Perl-compatible built-in Unicode alias normalization while - preserving deferred user-property precedence. The focused alias oracle - passes 16/16 on system Perl, JVM, and interpreter; unchanged - `regexp_unicode_prop.t` passes 1,110/1,110 on both execution backends. + - [x] Completed the built-in Unicode aliases exercised by + `regexp_unicode_prop.t` while preserving deferred user-property precedence. + The focused alias oracle passes 16/16 on system Perl, JVM, and interpreter; + unchanged `regexp_unicode_prop.t` passes 1,110/1,110 on both execution + backends. + - [x] Added a lossless, idempotent importer for Perl's generated TestProp + corpus. The focused importer test passes 66/66, two real generations are + byte-identical, system Perl executes 503,197 TAP, and JVM/interpreter both + execute 290,912 TAP with exact semantic parity and no timeout. + - [ ] Classify and close the 115,144 failures newly exposed by the lossless + generated `uniprops*.t` corpus; require JVM/interpreter plan and semantic + parity before marking Phase 3 complete. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) - [x] Preserved mixed executable-source provenance, nested dynamic callback @@ -413,18 +424,29 @@ matcher-specific timeouts on both execution backends. ### Next Steps -1. Complete the combined offset-map, regex-set, Joni safeguard, property - diagnostics/provenance, Unicode-alias, inline-`p`, and control-escape - integration while retaining each validated prerequisite head. -2. Integrate the lossless generated Unicode fixtures, then rerun the - forced-Joni corpus from the combined head with every child hard-bounded. -3. Capture clean JVM and interpreter 80-file baselines and compare every file - with PR 958 under the no-regression gate. -4. Classify residual engine-introspection exclusions, then queue the largest - remaining semantic cluster as a focused slice. -5. Keep the warning-free whole-unit-suite gate green with Joni as the default; - use explicit Java mode only to classify corpus regressions before Phase 5 - removes the legacy backend and selector. +1. Publish the validated lossless generated Unicode fixture importer on top of + PR #1016. Preserve the canonical corpus losslessly and retain hard per-child + bounds. The 300-second warm `uniprops04.t` attempt was contention-invalid; + repeat its performance measurement only on an idle host. +2. Rerun the forced-Joni 80-file corpus on JVM and interpreter from the combined + head. Save complete JSON and logs, publish the missing differential report, + and compare every file with both the Phase 0 result and PR 958 under the + no-regression gate. +3. Implement the largest residual semantic clusters from that report, including + the generated Unicode Age/alias/boundary groups and the accepted-invalid- + pattern and warning-policy losses in `reg_mesg.t`. Keep optimizer/debug- + transcript assertions explicitly classified rather than approximating Perl + internals. +4. Audit every `RegexPreprocessor` rule against the final ownership boundary. + Move matcher semantics into Joni, retain only source-policy scanning, delete + Java-only rewrites and compiled-pattern variants, and remove the temporary + Java backend selector after the performance gate passes. +5. Reconcile `docs/reference/feature-matrix.md` with the final corpus: replace + stale Unicode limitations, add any still-missing regex features, and link + each limitation to a reducer or explicit optimizer/debug exclusion. +6. Run unchanged CPAN consumers, the direct/thread release matrix, packaging + and license checks, and warning-free `make`; then rebase each focused PR and + require green Ubuntu and Windows CI. ### Open Questions and Blockers diff --git a/dev/import-perl5/README.md b/dev/import-perl5/README.md index f8942caa08..99c5a8230c 100644 --- a/dev/import-perl5/README.md +++ b/dev/import-perl5/README.md @@ -66,6 +66,7 @@ Main synchronization script that imports files from perl5/ based on config.yaml. **Features:** - Copies individual files or entire directories +- Generates pinned-source test artifacts such as `unicore/TestProp.pl` - Applies patches automatically - Creates necessary directories - Validates sources exist @@ -80,6 +81,34 @@ perl dev/import-perl5/sync.pl --help Protected targets (`protected: true` in YAML) are always excluded from bulk directory rsync using the **full** config list, even when `--only` is used. +#### Generated Unicode property fixture + +The upstream `uniprops01.t` through `uniprops10.t` wrappers load +`perl5_t/lib/unicore/TestProp.pl`. Perl does not commit that file: its build +generates it from `perl5/lib/unicore/mktables` and the pinned Unicode database. +PerlOnJava therefore generates a losslessly split fixture explicitly: + +```bash +perl dev/import-perl5/sync.pl --only TestProp.pl +``` + +Generation runs in a temporary copy of `perl5/lib/unicore`, so it does not +modify the pinned source tree. The importer retains every canonical assertion, +splits the ten existing `TESTCHUNK` sections into ignored `TestProp-01.pl` +through `TestProp-10.pl` files, and installs a small `TestProp.pl` dispatcher. +Each unchanged upstream wrapper consequently parses only its selected section +instead of parsing the complete 12 MB corpus before applying its chunk guard. +Running the dispatcher without `TESTCHUNK` still evaluates all sections in +canonical order. + +The dispatcher embeds the canonical SHA-256 and per-chunk call counts. The +dispatcher and all chunks remain ignored with the rest of `perl5_t/` and are +not included in the JAR or distribution. A clean checkout obtains them by +running the command above (a full unfiltered sync also runs this row). Unrelated +filtered syncs neither generate nor remove them. Missing Unicode inputs or an +incompatible host Perl make the sync fail with an actionable diagnostic rather +than silently leaving the ten wrappers unavailable. + ### add_module.pl Interactive tool to add new modules to the sync configuration. diff --git a/dev/import-perl5/config.yaml b/dev/import-perl5/config.yaml index beea16f400..51138f3246 100644 --- a/dev/import-perl5/config.yaml +++ b/dev/import-perl5/config.yaml @@ -3,7 +3,8 @@ # Each entry specifies: # source: Path relative to project root (typically perl5/...) # target: Path relative to project root where file/dir should be copied -# type: 'file' (default) or 'directory' for bulk directory copies +# type: 'file' (default), 'directory' for bulk copies, or a documented +# generated type handled by sync.pl (currently TestProp.pl only) # patch: Optional patch file (relative to dev/import-perl5/patches/) # protected: Optional boolean (true/false) - if true, existing file won't be overwritten # @@ -94,9 +95,12 @@ imports: target: perl5_t/Porting type: directory - # NOTE: perl5_t/lib directory should exist (for opendir tests) but remain empty - # to avoid module loading conflicts. PerlOnJava uses src/main/perl/lib instead. - # The sync.pl script will create an empty perl5_t/lib directory if needed. + # perl5_t/lib otherwise remains empty to avoid module loading conflicts. + # TestProp.pl is generated from the pinned Unicode inputs because upstream + # does not store this build artifact in the perl5 source tree. + - source: perl5/lib/unicore/mktables + target: perl5_t/lib/unicore/TestProp.pl + type: generated_unicode_testprop # Bulk import all core test files from perl5/t/ to perl5_t/t/ # This creates a complete Perl 5 test environment under perl5_t/ diff --git a/dev/import-perl5/sync.pl b/dev/import-perl5/sync.pl index c91b7a3785..7e459af9dc 100755 --- a/dev/import-perl5/sync.pl +++ b/dev/import-perl5/sync.pl @@ -16,7 +16,9 @@ use File::Path qw(make_path); use File::Copy qw(copy); use File::Spec; -use Cwd qw(abs_path); +use File::Temp qw(tempdir tempfile); +use Cwd qw(abs_path getcwd); +use Digest::SHA qw(sha256_hex); # Simple YAML parser for our specific needs sub parse_yaml { @@ -140,6 +142,249 @@ sub copy_directory { return 1; } +# Split the canonical generated corpus without dropping or reordering a byte of +# any TESTCHUNK section. The small dispatcher retains the canonical lexical +# helper scope and evals only the selected section. Leading newlines in each +# chunk preserve canonical caller line numbers in TAP diagnostics. +sub split_unicode_testprop { + my ($canonical) = @_; + my @markers; + while ($canonical + =~ /^if \(!\$::TESTCHUNK or \$::TESTCHUNK == (\d+)\) \{\s*$/mg) + { + push @markers, { number => 0 + $1, start => $-[0] }; + } + unless (join(',', map { $_->{number} } @markers) eq join(',', 1 .. 10)) { + die "Generated TestProp.pl has unexpected TESTCHUNK order: " + . (@markers ? join(',', map { $_->{number} } @markers) : '(none)') + . "\n"; + } + + pos($canonical) = $markers[-1]{start}; + my $finished_start; + if ($canonical =~ /^Finished\(\);\s*$/mg) { + $finished_start = $-[0]; + } + die "Generated TestProp.pl is missing Finished() after TESTCHUNK 10\n" + unless defined $finished_start; + + my $preamble = substr($canonical, 0, $markers[0]{start}); + my $finish = substr($canonical, $finished_start); + my @sections; + for my $index (0 .. $#markers) { + my $end = $index == $#markers + ? $finished_start + : $markers[$index + 1]{start}; + my $section = substr($canonical, $markers[$index]{start}, + $end - $markers[$index]{start}); + my $line_offset = substr($canonical, 0, $markers[$index]{start}) + =~ tr/\n//; + my %counts; + $counts{$1}++ while $section + =~ /^\s+(Expect|Error|Test_GCB|Test_SB|Test_LB|Test_WB)\(/mg; + push @sections, { + number => $markers[$index]{number}, + source => $section, + padded => ("\n" x $line_offset) . $section, + line_offset => $line_offset, + counts => \%counts, + }; + } + + my $reconstructed = $preamble . join('', map { $_->{source} } @sections) + . $finish; + die "Internal TestProp.pl split changed canonical content\n" + unless $reconstructed eq $canonical; + + my $canonical_sha = sha256_hex($canonical); + my @manifest = ( + "\n# PerlOnJava lossless TESTCHUNK dispatcher\n", + "# Canonical pinned mktables SHA-256: $canonical_sha\n", + ); + for my $section (@sections) { + my @counts = map { "$_=$section->{counts}{$_}" } + sort keys %{$section->{counts}}; + push @manifest, "# TESTCHUNK $section->{number}: " + . join(', ', @counts) . "\n"; + } + push @manifest, <<'DISPATCHER'; +my $__poj_testprop_dir = __FILE__; +$__poj_testprop_dir =~ s{[^/\\]+\z}{}; +my @__poj_testprop_chunks = !$::TESTCHUNK + ? (1 .. 10) + : ($::TESTCHUNK >= 1 && $::TESTCHUNK <= 10 ? ($::TESTCHUNK) : ()); +for my $__poj_testprop_chunk (@__poj_testprop_chunks) { + my $__poj_testprop_path = sprintf '%sTestProp-%02d.pl', + $__poj_testprop_dir, $__poj_testprop_chunk; + open my $__poj_testprop_fh, '<', $__poj_testprop_path + or die "Cannot load $__poj_testprop_path: $!\n"; + local $/; + my $__poj_testprop_source = <$__poj_testprop_fh>; + close $__poj_testprop_fh + or die "Cannot close $__poj_testprop_path: $!\n"; + my $__poj_testprop_completed = eval($__poj_testprop_source . "\n; 1;"); + my $__poj_testprop_error = $@; + unless ($__poj_testprop_completed) { + $__poj_testprop_error = "section eval returned false without an error\n" + unless length $__poj_testprop_error; + die "Cannot evaluate $__poj_testprop_path: $__poj_testprop_error"; + } +} +DISPATCHER + my $dispatcher = $preamble . join('', @manifest) . $finish; + return ($dispatcher, \@sections, $canonical_sha); +} + +# Generate the Unicode property test fixture from a pristine copy of the +# pinned source data. Upstream deliberately does not commit TestProp.pl; its +# normal build creates it with lib/unicore/mktables -maketest. +sub generate_unicode_testprop { + my ($generator_relative, $target, $project_root) = @_; + my $generator = File::Spec->catfile($project_root, $generator_relative); + my $unicode_source = dirname($generator); + my @required = ( + $generator, + File::Spec->catfile($unicode_source, 'version'), + File::Spec->catfile($unicode_source, 'UnicodeData.txt'), + ); + for my $required (@required) { + unless (-f $required) { + warn " ERROR: Cannot generate Unicode TestProp.pl; missing pinned " + . "generation prerequisite: $required\n" + . " Restore perl5/lib/unicore from the pinned perl5 source " + . "before running this sync.\n\n"; + return 0; + } + } + + my $temporary = tempdir('perlonjava-testprop-XXXXXX', TMPDIR => 1, CLEANUP => 1); + my $temporary_unicode = File::Spec->catdir($temporary, 'unicore'); + make_path($temporary_unicode) or do { + warn " ERROR: Cannot create temporary Unicode generation directory: $!\n\n"; + return 0; + }; + unless (copy_directory($unicode_source, $temporary_unicode, + $project_root, [], [])) { + warn " ERROR: Cannot copy pinned Unicode data for TestProp.pl generation.\n\n"; + return 0; + } + + my $generated = File::Spec->catfile($temporary_unicode, 'TestProp.pl'); + my $original_dir = getcwd(); + unless (chdir $project_root) { + warn " ERROR: Cannot enter project root for TestProp.pl generation: $!\n\n"; + return 0; + } + print " Generating with pinned Unicode data: $generator_relative\n"; + my $result = system($^X, $generator_relative, + '-C', $temporary_unicode, + '-T', $generated, + '-q'); + my $generation_error = $!; + unless (chdir $original_dir) { + die "sync.pl: cannot restore working directory '$original_dir': $!\n"; + } + if ($result != 0) { + my $exit = $result == -1 + ? "could not start: $generation_error" + : ($result & 127) + ? "terminated by signal " . ($result & 127) + : "exit code " . ($result >> 8); + warn " ERROR: Unicode TestProp.pl generation failed ($exit).\n" + . " Ensure the pinned perl5/lib/unicore data is complete and the " + . "host Perl can run mktables.\n\n"; + return 0; + } + unless (-s $generated) { + warn " ERROR: Unicode generator completed without producing TestProp.pl.\n\n"; + return 0; + } + + open my $generated_fh, '<', $generated or do { + warn " ERROR: Cannot inspect generated TestProp.pl: $!\n\n"; + return 0; + }; + local $/; + my $contents = <$generated_fh>; + close $generated_fh; + my ($dispatcher, $sections, $canonical_sha); + eval { + ($dispatcher, $sections, $canonical_sha) + = split_unicode_testprop($contents); + 1; + } or do { + my $error = $@ || 'unknown split error'; + chomp $error; + warn " ERROR: Cannot split generated TestProp.pl: $error\n\n"; + return 0; + }; + + my $target_dir = dirname($target); + unless (-d $target_dir) { + make_path($target_dir) or do { + warn " ERROR: Cannot create generated fixture directory: $!\n\n"; + return 0; + }; + } + my @outputs; + my ($volume, $directories, $base) = File::Spec->splitpath($target); + $base =~ s/\.pl\z//; + for my $section (@$sections) { + my $chunk_base = sprintf '%s-%02d.pl', $base, $section->{number}; + my $chunk_target = File::Spec->catpath($volume, $directories, + $chunk_base); + push @outputs, [$chunk_target, $section->{padded}]; + } + # Publish the dispatcher last so readers never see it before all ten chunk + # names exist. Stage every output completely before replacing any member of + # the current family; a write failure therefore leaves that family intact. + push @outputs, [$target, $dispatcher]; + my @staged; + for my $output (@outputs) { + my ($path, $source) = @$output; + my ($output_fh, $staged_path); + eval { + ($output_fh, $staged_path) = tempfile( + '.TestProp-sync-XXXXXX', DIR => $target_dir, UNLINK => 0 + ); + 1; + } or do { + my $error = $@ || 'unknown temporary-file error'; + chomp $error; + warn " ERROR: Cannot stage generated fixture $path: $error\n\n"; + unlink $_->[0] for @staged; + return 0; + }; + print {$output_fh} $source or do { + warn " ERROR: Cannot stage generated fixture $path: $!\n\n"; + close $output_fh; + unlink $staged_path; + unlink $_->[0] for @staged; + return 0; + }; + close $output_fh or do { + warn " ERROR: Cannot close staged generated fixture $path: $!\n\n"; + unlink $staged_path; + unlink $_->[0] for @staged; + return 0; + }; + push @staged, [$staged_path, $path]; + } + while (my $staged = shift @staged) { + my ($staged_path, $path) = @$staged; + unless (rename $staged_path, $path) { + warn " ERROR: Cannot publish generated fixture $path: $!\n\n"; + unlink $staged_path; + unlink $_->[0] for @staged; + return 0; + } + } + print " Installed lossless generated fixture dispatcher and 10 chunks: " + . File::Spec->abs2rel($target, $project_root) + . " (canonical SHA-256 $canonical_sha)\n"; + return 1; +} + sub usage { print <<'USAGE'; PerlOnJava perl5 import sync — see dev/import-perl5/config.yaml @@ -264,7 +509,13 @@ sub main { print "Processing: $import->{source}\n"; # Check if source exists - if ($type eq 'directory') { + if ($type eq 'generated_unicode_testprop') { + unless (generate_unicode_testprop($import->{source}, $target, $project_root)) { + $error_count++; + next; + } + } + elsif ($type eq 'directory') { unless (-d $source) { warn " ERROR: Source directory not found: $source\n\n"; $error_count++; @@ -342,7 +593,8 @@ sub main { $success_count++; } - # Create empty perl5_t/lib directory (needed for opendir tests but must stay empty) + # Create perl5_t/lib for opendir tests. It remains empty except for + # explicitly configured generated test fixtures such as unicore/TestProp.pl. my $lib_dir = File::Spec->catdir($project_root, 'perl5_t', 'lib'); unless (-d $lib_dir) { print "Creating empty perl5_t/lib directory...\n"; @@ -362,4 +614,6 @@ sub main { } } -main(); +main() unless caller; + +1; diff --git a/dev/import-perl5/t/generated_unicode_testprop.t b/dev/import-perl5/t/generated_unicode_testprop.t new file mode 100644 index 0000000000..a2284a9047 --- /dev/null +++ b/dev/import-perl5/t/generated_unicode_testprop.t @@ -0,0 +1,203 @@ +use strict; +use warnings; +use Test::More; +use File::Path qw(make_path); +use File::Spec; +use File::Temp qw(tempdir); +use Digest::SHA qw(sha256_hex); + +my $sync = File::Spec->catfile('dev', 'import-perl5', 'sync.pl'); +do "./$sync" or die "Could not load $sync: $@ $!"; + +my $root = tempdir(CLEANUP => 1); +my $unicode = File::Spec->catdir($root, 'perl5', 'lib', 'unicore'); +make_path($unicode); + +my $generator_relative = File::Spec->catfile('perl5', 'lib', 'unicore', 'mktables'); +my $generator = File::Spec->catfile($root, $generator_relative); +open my $generator_fh, '>', $generator or die "Cannot create fake generator: $!"; +print {$generator_fh} <<'GENERATOR'; +use strict; +use warnings; +use File::Spec; +my ($directory, $target); +while (@ARGV) { + my $argument = shift; + $directory = shift if $argument eq '-C'; + $target = shift if $argument eq '-T'; +} +die "missing generation arguments\n" unless defined $directory && defined $target; +open my $output, '>', $target or die "Cannot create $target: $!"; +print {$output} <<'PREAMBLE'; +# generated fixture +my $helper_lexical = 1; +sub Expect { $main::testprop_expects += $helper_lexical } +sub Error { $main::testprop_errors += $helper_lexical } +sub Test_GCB { $main::testprop_breaks += $helper_lexical } +sub Test_SB { $main::testprop_breaks += $helper_lexical } +sub Test_LB { $main::testprop_breaks += $helper_lexical } +sub Test_WB { $main::testprop_breaks += $helper_lexical } +sub Finished { $main::testprop_finished++ } +PREAMBLE +for my $chunk (1 .. 10) { + print {$output} "if (!\$::TESTCHUNK or \$::TESTCHUNK == $chunk) {\n"; + print {$output} " \$main::testprop_chunks .= '$chunk,';\n"; + if ($chunk <= 4) { + for my $index (1 .. 600) { + my $suffix = " # chunk-$chunk-expect-$index"; + $suffix .= " regression-$index" + if $chunk == 1 && $index <= 16; + print {$output} " Expect();$suffix\n"; + } + print {$output} " Error(); # chunk-$chunk-error-$_\n" for 1 .. 300; + } + elsif ($chunk == 5) { + print {$output} " Test_GCB();\n" for 1 .. 120; + print {$output} " Test_SB();\n" for 1 .. 80; + } + elsif ($chunk <= 9) { + print {$output} " Test_LB();\n" for 1 .. 80; + } + else { + print {$output} " Test_WB();\n" for 1 .. 80; + } + print {$output} "}\n"; +} +print {$output} "Finished();\n"; +close $output; +GENERATOR +close $generator_fh; + +for my $required (qw(version UnicodeData.txt)) { + my $path = File::Spec->catfile($unicode, $required); + open my $fh, '>', $path or die "Cannot create $path: $!"; + print {$fh} $required eq 'version' ? "17.0.0\n" : "0041;LATIN CAPITAL LETTER A\n"; + close $fh; +} + +my $target = File::Spec->catfile($root, 'perl5_t', 'lib', 'unicore', 'TestProp.pl'); +my @family = ($target, map { + File::Spec->catfile($root, 'perl5_t', 'lib', 'unicore', + sprintf('TestProp-%02d.pl', $_)) +} 1 .. 10); + +sub slurp { + my ($path) = @_; + open my $fh, '<', $path or die "Cannot read $path: $!"; + local $/; + my $contents = <$fh>; + close $fh; + return $contents; +} + +ok(generate_unicode_testprop($generator_relative, $target, $root), + 'generated import succeeds from pinned inputs'); +ok(-s $_, "generated fixture is installed: $_") for @family; +my %first_family = map { $_ => slurp($_) } @family; +my $first = $first_family{$target}; +like($first, qr/^# Canonical pinned mktables SHA-256: ([0-9a-f]{64})$/m, + 'dispatcher records its canonical generated-input hash'); +my ($recorded_canonical_sha) = $first + =~ /^# Canonical pinned mktables SHA-256: ([0-9a-f]{64})$/m; +my @reconstructed_sections; +for my $chunk (1 .. 10) { + like($first, qr/^# TESTCHUNK $chunk: /m, + "dispatcher records TESTCHUNK $chunk counts"); + my $path = $family[$chunk]; + my $source = $first_family{$path}; + my $marker = "if (!\$::TESTCHUNK or \$::TESTCHUNK == $chunk) {"; + my $start = index($source, $marker); + cmp_ok($start, '>=', 0, "chunk $chunk contains its canonical marker"); + is(index($source, 'TESTCHUNK == ', $start + length $marker), -1, + "chunk $chunk contains no later TESTCHUNK section"); + push @reconstructed_sections, substr($source, $start); +} +my ($canonical_preamble) = split /\n# PerlOnJava lossless TESTCHUNK dispatcher\n/, + $first, 2; +my $reconstructed = $canonical_preamble + . join('', @reconstructed_sections) . "Finished();\n"; +is(sha256_hex($reconstructed), $recorded_canonical_sha, + 'dispatcher and chunks reconstruct the canonical generated hash'); +my @expects = $reconstructed =~ /^\s+Expect\(/mg; +my @errors = $reconstructed =~ /^\s+Error\(/mg; +is(scalar @expects, 4 * 600, 'all canonical Expect calls are retained'); +is(scalar @errors, 4 * 300, 'all canonical Error calls are retained'); +like($reconstructed, qr/# chunk-2-expect-1$/m, + 'first ordered Expect call is retained'); +like($reconstructed, qr/# chunk-2-expect-600$/m, + 'last ordered Expect call is retained'); +like($reconstructed, qr/# chunk-2-error-1$/m, + 'first ordered Error call is retained'); +like($reconstructed, qr/# chunk-2-error-300$/m, + 'last ordered Error call is retained'); + +{ + local $::TESTCHUNK = 2; + local $::testprop_chunks = ''; + local $::testprop_expects = 0; + local $::testprop_errors = 0; + local $::testprop_breaks = 0; + local $::testprop_finished = 0; + my $loaded = do $target; + ok(defined $loaded, 'dispatcher executes one selected generated chunk') + or diag "dispatcher error: $@ $!"; + is($::testprop_chunks, '2,', 'dispatcher loads exactly the selected chunk'); + is($::testprop_expects, 600, + 'selected chunk eval sees canonical lexical helper scope'); + is($::testprop_errors, 300, 'selected chunk retains every Error call'); + is($::testprop_finished, 1, 'selected dispatch calls canonical Finished once'); +} +{ + local $::TESTCHUNK = 0; + local $::testprop_chunks = ''; + local $::testprop_expects = 0; + local $::testprop_errors = 0; + local $::testprop_breaks = 0; + local $::testprop_finished = 0; + my $loaded = do $target; + ok(defined $loaded, 'dispatcher executes all chunks when none is selected') + or diag "dispatcher error: $@ $!"; + is($::testprop_chunks, '1,2,3,4,5,6,7,8,9,10,', + 'unselected dispatch preserves canonical chunk order'); + is($::testprop_expects, 4 * 600, 'all-chunk dispatch retains all Expect calls'); + is($::testprop_errors, 4 * 300, 'all-chunk dispatch retains all Error calls'); + is($::testprop_breaks, 120 + 80 + 4 * 80 + 80, + 'all-chunk dispatch retains all boundary calls'); + is($::testprop_finished, 1, 'all-chunk dispatch calls canonical Finished once'); +} + +open my $damage_fh, '>', $target or die "Cannot alter $target: $!"; +print {$damage_fh} "damaged\n"; +close $damage_fh; +open my $chunk_damage_fh, '>', $family[7] + or die "Cannot alter $family[7]: $!"; +print {$chunk_damage_fh} "damaged chunk\n"; +close $chunk_damage_fh; +ok(generate_unicode_testprop($generator_relative, $target, $root), + 'repeated generation succeeds'); +my %second_family = map { $_ => slurp($_) } @family; +is_deeply(\%second_family, \%first_family, + 'repeated generation installs a byte-identical dispatcher and chunks'); + +unlink File::Spec->catfile($unicode, 'version') + or die "Cannot remove fake version prerequisite: $!"; +my $warning = ''; +{ + local $SIG{__WARN__} = sub { $warning .= join '', @_ }; + ok(!generate_unicode_testprop($generator_relative, $target, $root), + 'missing generation prerequisite fails'); +} +like($warning, qr/missing pinned generation prerequisite.*version/s, + 'missing prerequisite diagnostic names the absent input'); + +my $bad_order = join '', + "if (!\$::TESTCHUNK or \$::TESTCHUNK == 2) {\n}\n", + "if (!\$::TESTCHUNK or \$::TESTCHUNK == 1) {\n}\n", + (map { "if (!\$::TESTCHUNK or \$::TESTCHUNK == $_) {\n}\n" } 3 .. 10), + "Finished();\n"; +my $order_error = ''; +eval { split_unicode_testprop($bad_order); 1 } or $order_error = $@; +like($order_error, qr/unexpected TESTCHUNK order: 2,1,3,4,5,6,7,8,9,10/, + 'splitting rejects dropped or reordered canonical sections'); + +done_testing; diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index ca44b5028f..7c5dc0a117 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -394,7 +394,7 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Script and block values, script extensions, `Extended_Pictographic`, exact `Age`, and cumulative `In`/`Present_In` properties execute through Joni; remaining work is limited to uncommon Perl-only aliases and exact invalid-property diagnostics. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. The lossless generated `uniprops*.t` matrix currently passes 175,768/290,912 identically on JVM and interpreter; remaining work includes Perl's broad loose property/value alias spellings and boundary edge cases exposed by chunks 01–05. - ✅ **Extended Grapheme Clusters**: `\X` matches combining sequences and emoji ZWJ grapheme clusters. - ✅ **Embedded Code in Regex**: `(?{ code })`, optimistic callbacks `(*{ code })`, executable callback conditions, and `(??{ code })` run as lexical closures in Joni with provisional captures and backtracking unwind. Callback `local` frames follow matcher paths, and escaped loop control or `goto` stops at the callback pseudo-block boundary. `$^N` follows capture-close order independently of `$+`. - ✅ **Regex Debugging**: Lexically scoped `use/no re 'debug'` and `debugcolor` are supported, including runtime snapshot ownership. From 9c85fc33e9017fe074375c789b46ebc002035034 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:13:44 +0200 Subject: [PATCH 14/60] fix(regex): normalize Is property assignments (#1019) Squashed integration of PR #1019 at 39be7ed7e. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 63 +++++++++++++------ docs/reference/feature-matrix.md | 2 +- .../runtime/regex/UnicodeResolver.java | 45 ++++++++++++- .../regex/unicode_property_is_assignment.t | 18 ++++++ 4 files changed, 106 insertions(+), 22 deletions(-) create mode 100644 src/test/resources/unit/regex/unicode_property_is_assignment.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index ae2ab0be00..ee0613d1f1 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -237,10 +237,22 @@ The lossless generated `TestProp.pl` fixture now exposes ten previously gated assertions with zero timeouts or incomplete files, at 175,768 passing and 115,144 failing. PR 958 recorded zero plan for all ten files, so this is new coverage rather than a regression, but it reopens Phase 3's full-corpus exit -gate. Chunks 06–10 are complete except for 791 failures in chunk 05; chunks -01–04 require semantic clustering before Phase 3 can close. -The interpreter produces the exact same 175,768/290,912 split, with identical -per-file plans and no timeout or incomplete file. +gate. The interpreter produces the exact same 175,768/290,912 split, with +identical per-file plans and no timeout or incomplete file. Semantic-key +validation found that all 123,411 generated records +in chunks 05–10 retain literal UTF-8 boundary markers in both subjects and +patterns. Their 122,620 apparent passing assertions are therefore vacuous, +not evidence of boundary or folding parity. Chunks 01–04 contain the genuine +property and hand-written boundary evidence. An exact upstream-stage reducer +shows that lexical `use bytes` fails to replace the upgraded marker regex in a +byte subject (system Perl 4/4, JVM/interpreter 2/4); that byte-mode substitution +gap must be fixed before chunks 05–10 can become an authoritative boundary gate. + +Explicit `Is_*` property/value assignments now normalize before built-in +routing, including Perl's colon delimiter. The focused 12-assertion oracle +passes on system Perl, JVM, and interpreter. The generated corpus rises by +exactly 44,944 assertions to 220,712/290,912 on both execution backends, with +identical per-file counts and zero errors, timeouts, or incomplete files. Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while @@ -406,9 +418,16 @@ matcher-specific timeouts on both execution backends. corpus. The focused importer test passes 66/66, two real generations are byte-identical, system Perl executes 503,197 TAP, and JVM/interpreter both execute 290,912 TAP with exact semantic parity and no timeout. - - [ ] Classify and close the 115,144 failures newly exposed by the lossless - generated `uniprops*.t` corpus; require JVM/interpreter plan and semantic - parity before marking Phase 3 complete. + - [x] Classified all 115,144 failures newly exposed by the lossless generated + `uniprops*.t` corpus, including the cross-cutting invalid boundary-harness + evidence in chunks 05–10. + - [x] Normalized explicit `Is_*` property/value assignments and the colon + delimiter (PR #1019), gaining exactly 44,944 generated assertions on both + execution backends without changing any plan. + - [ ] Fix byte-mode substitution of upgraded marker regexes so chunks 05–10 + exercise real boundary subjects, then close the property and boundary + failures with exact JVM/interpreter plan and semantic parity before marking + Phase 3 complete. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) - [x] Preserved mixed executable-source provenance, nested dynamic callback @@ -424,27 +443,31 @@ matcher-specific timeouts on both execution backends. ### Next Steps -1. Publish the validated lossless generated Unicode fixture importer on top of - PR #1016. Preserve the canonical corpus losslessly and retain hard per-child - bounds. The 300-second warm `uniprops04.t` attempt was contention-invalid; - repeat its performance measurement only on an idle host. -2. Rerun the forced-Joni 80-file corpus on JVM and interpreter from the combined +1. Finalize PR #1019 after its tracker correction, then integrate the validated + fatal-Joni-syntax commit `d2fd88096` (+259 raw `reg_mesg.t` passes; warning- + free `make`) and publish it as the next stacked slice. +2. Fix lexical `use bytes` substitution when an upgraded marker regex matches a + byte subject, without patching generated fixtures. Regenerate the lossless + corpus and prove that chunks 05–10 no longer match literal boundary markers + before treating any boundary or folding result as evidence. Keep the + separately reduced eval byte-source UTF-8 decoding fix as an independent + runtime parity slice. +3. Implement the remaining property clusters in measured order: Block, + Script/Script_Extensions, Numeric_Value, Joining_Group, General_Category, + break-property values, and Age/In/Present_In. Preserve pinned Perl 5.44 + acceptance and rejection semantics rather than inheriting host ICU breadth. +4. Rerun the forced-Joni 80-file corpus on JVM and interpreter from the combined head. Save complete JSON and logs, publish the missing differential report, and compare every file with both the Phase 0 result and PR 958 under the no-regression gate. -3. Implement the largest residual semantic clusters from that report, including - the generated Unicode Age/alias/boundary groups and the accepted-invalid- - pattern and warning-policy losses in `reg_mesg.t`. Keep optimizer/debug- - transcript assertions explicitly classified rather than approximating Perl - internals. -4. Audit every `RegexPreprocessor` rule against the final ownership boundary. +5. Audit every `RegexPreprocessor` rule against the final ownership boundary. Move matcher semantics into Joni, retain only source-policy scanning, delete Java-only rewrites and compiled-pattern variants, and remove the temporary Java backend selector after the performance gate passes. -5. Reconcile `docs/reference/feature-matrix.md` with the final corpus: replace +6. Reconcile `docs/reference/feature-matrix.md` with the final corpus: replace stale Unicode limitations, add any still-missing regex features, and link each limitation to a reducer or explicit optimizer/debug exclusion. -6. Run unchanged CPAN consumers, the direct/thread release matrix, packaging +7. Run unchanged CPAN consumers, the direct/thread release matrix, packaging and license checks, and warning-free `make`; then rebase each focused PR and require green Ubuntu and Windows CI. diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index 7c5dc0a117..3e3e3ccb02 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -394,7 +394,7 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. The lossless generated `uniprops*.t` matrix currently passes 175,768/290,912 identically on JVM and interpreter; remaining work includes Perl's broad loose property/value alias spellings and boundary edge cases exposed by chunks 01–05. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. The lossless generated `uniprops*.t` matrix currently passes 220,712/290,912 identically on JVM and interpreter, but chunks 01–04 still expose broad loose property/value alias gaps and chunks 05–10 are not yet valid boundary evidence because byte-mode substitution does not normalize upgraded boundary-marker regexes against byte subjects. - ✅ **Extended Grapheme Clusters**: `\X` matches combining sequences and emoji ZWJ grapheme clusters. - ✅ **Embedded Code in Regex**: `(?{ code })`, optimistic callbacks `(*{ code })`, executable callback conditions, and `(??{ code })` run as lexical closures in Joni with provisional captures and backtracking unwind. Callback `local` frames follow matcher paths, and escaped loop control or `goto` stops at the callback pseudo-block boundary. `$^N` follows capture-close order independently of `$+`. - ✅ **Regex Debugging**: Lexically scoped `use/no re 'debug'` and `debugcolor` are supported, including runtime snapshot ownership. diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index f1b1142a34..8957f8feb7 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -699,6 +699,7 @@ private static String translateUnicodeProperty(String property, boolean negated, property = property.substring(1).trim(); negated = !negated; } + property = normalizePerlIsPropertyAssignment(property); if (property.startsWith("utf8::")) { String userPropertyName = property.substring("utf8::".length()); if (!userPropertyName.matches("[A-Za-z_][A-Za-z0-9_]*")) { @@ -933,7 +934,8 @@ static boolean isUserDefinedPropertyName(String property) { } static boolean isPerlBuiltInPropertyAlias(String property) { - return resolvePerlBuiltInPropertyAlias(property) != null; + return !normalizePerlIsPropertyAssignment(property).equals(property) + || resolvePerlBuiltInPropertyAlias(property) != null; } private static UnicodeSet resolvePerlBuiltInPropertyAlias(String property) { @@ -992,6 +994,47 @@ private static String loosePropertyName(String value) { return normalized.toString(); } + /** + * Perl accepts an {@code Is_} prefix before property names in explicit + * property/value forms, and accepts a colon in place of the equals sign. + * The prefix is distinct from a bare {@code IsFoo} user property because + * this normalization applies only when a value delimiter is present. + */ + private static String normalizePerlIsPropertyAssignment(String property) { + int delimiter = property.indexOf('='); + if (delimiter < 0) { + for (int i = 1; i < property.length() - 1; i++) { + if (property.charAt(i) == ':' + && property.charAt(i - 1) != ':' + && property.charAt(i + 1) != ':') { + delimiter = i; + break; + } + } + } + if (delimiter <= 0 || delimiter == property.length() - 1) { + return property; + } + + String name = property.substring(0, delimiter); + String looseName = loosePropertyName(name); + if (!looseName.startsWith("is") || looseName.length() == 2) { + return property; + } + + int nameStart = 2; + while (nameStart < name.length()) { + char separator = name.charAt(nameStart); + if (!Character.isWhitespace(separator) + && separator != '-' && separator != '_') { + break; + } + nameStart++; + } + if (nameStart >= name.length()) return property; + return name.substring(nameStart) + "=" + property.substring(delimiter + 1); + } + private static UnicodeSet unicodePropertyValueSet(int property, String alias) { int value = unicodePropertyValue(property, alias); if (value < 0) return null; diff --git a/src/test/resources/unit/regex/unicode_property_is_assignment.t b/src/test/resources/unit/regex/unicode_property_is_assignment.t new file mode 100644 index 0000000000..d1ba6f8f90 --- /dev/null +++ b/src/test/resources/unit/regex/unicode_property_is_assignment.t @@ -0,0 +1,18 @@ +use strict; +use warnings; +use Test::More tests => 12; + +ok("\x{1E900}" =~ /\p{Is_Block=Adlam}/, 'Is_Block property prefix'); +ok("\x{1E900}" =~ /\p{Is_Blk=Adlam}/, 'Is_Blk property prefix'); +ok("\x{1E900}" =~ /\p{Is_Script=Adlam}/, 'Is_Script property prefix'); +ok("\x{1E900}" =~ /\p{Is_Sc=Adlm}/, 'Is_Sc property prefix'); +ok('A' =~ /\p{Is_General_Category=Uppercase_Letter}/, + 'Is_General_Category property prefix'); +ok('A' =~ /\p{Is_Gc=Lu}/, 'Is_Gc property prefix'); +ok('A' =~ /\p{Is_Age=1.1}/, 'Is_Age exact property prefix'); +ok('A' =~ /\p{Is_In=1.1}/, 'Is_In cumulative property prefix'); +ok('A' =~ /\p{Is_Present_In=1.1}/, 'Is_Present_In cumulative prefix'); +ok('1' =~ /\p{Is_Numeric_Value=1}/, 'Is_Numeric_Value prefix'); +ok("\x{0301}" =~ /\p{Is_Canonical_Combining_Class:230}/, + 'Is_ property with colon delimiter'); +ok("\r" =~ /\p{Is_Line_Break=CR}/, 'Is_Line_Break prefix'); From 2ec453e466bb862ac93b6e8fdcede0679d05a925 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:13:44 +0200 Subject: [PATCH 15/60] fix(regex): honor UTF-8 in byte-backed eval source (#1020) Squashed integration of PR #1020 at c99737485. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 17 +++++-- docs/reference/feature-matrix.md | 11 +++-- .../backend/bytecode/EvalStringHandler.java | 14 +++++- .../frontend/parser/StringParser.java | 25 +++++++++- .../runtime/runtimetypes/RuntimeCode.java | 37 +++++++++++---- .../unit/regex/eval_byte_source_utf8.t | 46 +++++++++++++++++++ 6 files changed, 131 insertions(+), 19 deletions(-) create mode 100644 src/test/resources/unit/regex/eval_byte_source_utf8.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index ee0613d1f1..58eba7dba9 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -438,7 +438,15 @@ matcher-specific timeouts on both execution backends. the final owning regex scope exits (test 307). - [x] Resolved failed-path `$^N`/`$+` tests 85-86 on JVM and interpreter. - [x] Classified tests 444-448 as optimizer/debug-transcript exclusions. + - [x] Decoded byte-backed eval source according to lexical `use utf8`, + including pragmas activated inside the source, while preserving `no utf8` + byte semantics and fatal malformed-UTF-8 diagnostics. The focused oracle + passes 7/7 on system Perl, JVM, interpreter, and the direct JVM eval + compiler. - [ ] Phase 5: Remove the Java matching backend + - [x] Retired the unreachable top-level `(*PRUNE)` text rewrite after native + Joni control-verb gates passed under default and forced-Java policy + (`5760874e4`; 316 preprocessor lines removed). - [ ] Phase 6: Integration and release ### Next Steps @@ -449,13 +457,16 @@ matcher-specific timeouts on both execution backends. 2. Fix lexical `use bytes` substitution when an upgraded marker regex matches a byte subject, without patching generated fixtures. Regenerate the lossless corpus and prove that chunks 05–10 no longer match literal boundary markers - before treating any boundary or folding result as evidence. Keep the - separately reduced eval byte-source UTF-8 decoding fix as an independent - runtime parity slice. + before treating any boundary or folding result as evidence. Retain the now- + completed eval byte-source UTF-8 oracle as an independent runtime regression + gate; it is not the generated-corpus marker fix. 3. Implement the remaining property clusters in measured order: Block, Script/Script_Extensions, Numeric_Value, Joining_Group, General_Category, break-property values, and Age/In/Present_In. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. + Close the independently reduced Joni syntax gaps for Python-style named + groups, alpha assertion aliases, underscored numeric escapes, and braced + octal parsing with focused standard-Perl gates. 4. Rerun the forced-Joni 80-file corpus on JVM and interpreter from the combined head. Save complete JSON and logs, publish the missing differential report, and compare every file with both the Phase 0 result and PR 958 under the diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index 3e3e3ccb02..f5a0e7c8eb 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -371,19 +371,22 @@ my @copy = @{$z}; # ERROR - ✅ **Inline comments**: `(?#comment)` in regex is implemented. - ✅ **caret modifier**: `(?^` embedded pattern-match modifier, shorthand equivalent to "d-imnsx". - ✅ **\b inside character class**: `[\b]` is supported in regex. -- ✅ **\b{gcb} \B{gcb}**: Boundary assertions. +- 🟡 **Unicode boundary assertions**: `\b{gcb}`, `\B{gcb}`, and the `wb`/`lb`/`sb` forms are recognized, but the genuine hand-written generated-corpus preamble still has 25 semantic failures; comprehensive generated coverage is blocked by byte-mode marker normalization. - ✅ **Variable Interpolation in Regex**: Features like `${var}` for embedding variables. - ✅ **Non-capturing groups**: `(?:...)` is implemented. - ✅ **Named Capture Groups**: Defining named capture groups using `(?...)` or `(?'name'...)` is supported. - ✅ **Backreferences to Named Groups**: Using `\k` or `\g{name}` for backreferences to named groups is supported. - ✅ **Relative Backreferences**: Using `\g{-n}` for relative backreferences. -- ✅ **Unicode Properties**: Matching with `\p{...}` and `\P{...}` (e.g., `\p{L}` for letters). -- ✅ **Unicode Properties**: Add regex properties supported by Perl but missing in Java regex. +- ✅ **Basic Unicode Properties**: Common `\p{...}` and `\P{...}` forms such as `\p{L}` execute through Joni. +- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block/Age aliases execute through Joni, but the generated corpus still exposes loose property/value aliases and pinned acceptance/rejection gaps. - ✅ **Possessive Quantifiers**: Quantifiers like `*+`, `++`, `?+`, and `{n,m}+`, which disable backtracking, are supported. - ✅ **Atomic Grouping**: Use of `(?>...)` for atomic groups is supported. - ✅ **`\K` assertion**: Keep left — in `s///`, text before `\K` is preserved; match variables reflect only the portion after `\K`. - ✅ **Preprocessor**: `\Q`, `\L`, `\U`, `\l`, `\u`, `\E` are preprocessed in regex. - ✅ **Overloading**: `qr` overloading is implemented. See also [overload pragma](#pragmas). +- ❌ **Python-style named groups**: `(?P...)` and `(?P=name)` are accepted by Perl but are not yet parsed by Joni. +- ❌ **Alpha assertion aliases**: `(*pla:...)`, `(*plb:...)`, `(*nla:...)`, `(*nlb:...)`, and `(*atomic:...)` are not yet parsed by Joni. +- ❌ **Underscored numeric regex escapes**: Perl spellings such as `\x{0_0_4_1}` and `\o{0_0_1_0_1}` are not yet parsed by Joni; braced octal syntax also needs complete validation and diagnostics. - ✅ **Dynamically-scoped regex variables**: Provisional captures, `$^R`, `$^N`, match positions, and callback locals follow matcher paths and unwind on backtracking. - ✅ **Recursive and Dynamic Patterns**: `(?R)`, `(?0)`, and runtime `(??{ code })` execute through Joni. Dynamic expressions may return strings or `qr//` values, nested alternatives participate in outer backtracking without changing outer grouping or capture numbering, and callback and pure-pattern recursion have engine-owned depth ceilings. @@ -395,7 +398,7 @@ my @copy = @{$z}; # ERROR - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. - 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. The lossless generated `uniprops*.t` matrix currently passes 220,712/290,912 identically on JVM and interpreter, but chunks 01–04 still expose broad loose property/value alias gaps and chunks 05–10 are not yet valid boundary evidence because byte-mode substitution does not normalize upgraded boundary-marker regexes against byte subjects. -- ✅ **Extended Grapheme Clusters**: `\X` matches combining sequences and emoji ZWJ grapheme clusters. +- 🟡 **Extended Grapheme Clusters**: Focused `\X` combining-sequence and emoji-ZWJ cases pass, but comprehensive generated UAX verification remains blocked by byte-mode boundary-marker normalization. - ✅ **Embedded Code in Regex**: `(?{ code })`, optimistic callbacks `(*{ code })`, executable callback conditions, and `(??{ code })` run as lexical closures in Joni with provisional captures and backtracking unwind. Callback `local` frames follow matcher paths, and escaped loop control or `goto` stops at the callback pseudo-block boundary. `$^N` follows capture-close order independently of `$+`. - ✅ **Regex Debugging**: Lexically scoped `use/no re 'debug'` and `debugcolor` are supported, including runtime snapshot ownership. - ✅ **Runtime Regex Evaluation**: `use re 'eval'` controls whether interpolated patterns containing eval groups may compile. Admitted runtime source is compiled into lexical callback closures and preserves its package, visible lexical cells, Unicode or byte source type, default regex modifiers, and match-once state. Literal callbacks, interpolated `qr//` values (including local, referenced, and tied arrays), raw runtime eval groups, callback conditions, and standalone `(?(DEFINE)...)` containers may be composed in one Joni pattern; dynamic callbacks may return further admitted executable source. diff --git a/src/main/java/org/perlonjava/backend/bytecode/EvalStringHandler.java b/src/main/java/org/perlonjava/backend/bytecode/EvalStringHandler.java index 589de314c9..e1b268421c 100644 --- a/src/main/java/org/perlonjava/backend/bytecode/EvalStringHandler.java +++ b/src/main/java/org/perlonjava/backend/bytecode/EvalStringHandler.java @@ -13,6 +13,7 @@ import org.perlonjava.frontend.semantic.ScopedSymbolTable; import org.perlonjava.runtime.operators.WarnDie; import org.perlonjava.runtime.perlmodule.BHooksEndOfScope; +import org.perlonjava.runtime.perlmodule.Strict; import org.perlonjava.runtime.runtimetypes.*; import java.util.ArrayList; @@ -236,7 +237,14 @@ private static RuntimeList evalStringList(String perlCode, // Step 1: Clear $@ at start of eval GlobalVariable.setGlobalVariable("main::@", ""); - if (isEvalbytes && RuntimeCode.shouldDecodeEvalbytesUtf8Source(perlCode)) { + int inheritedStrictOptions = siteStrictOptions >= 0 + ? siteStrictOptions + : currentCode != null ? currentCode.strictOptions : 0; + boolean byteStringUtf8Source = !isEvalbytes + && sourceType == RuntimeScalarType.BYTE_STRING + && (inheritedStrictOptions & Strict.HINT_UTF8) != 0; + if ((isEvalbytes && RuntimeCode.shouldDecodeEvalbytesUtf8Source(perlCode)) + || byteStringUtf8Source) { perlCode = RuntimeCode.decodeEvalbytesUtf8Source(perlCode); } @@ -254,7 +262,9 @@ private static RuntimeList evalStringList(String perlCode, CompilerOptions opts = new CompilerOptions(); opts.fileName = evalFileName; - configureEvalSourceOptions(opts, perlCode, sourceType, isEvalbytes); + configureEvalSourceOptions(opts, perlCode, + byteStringUtf8Source ? RuntimeScalarType.STRING : sourceType, + isEvalbytes); ScopedSymbolTable symbolTable = new ScopedSymbolTable(); // Add standard variables that are always available in eval context. diff --git a/src/main/java/org/perlonjava/frontend/parser/StringParser.java b/src/main/java/org/perlonjava/frontend/parser/StringParser.java index d5dca10038..fc53b5022a 100644 --- a/src/main/java/org/perlonjava/frontend/parser/StringParser.java +++ b/src/main/java/org/perlonjava/frontend/parser/StringParser.java @@ -240,8 +240,13 @@ public static ParsedString parseRawStringWithDelimiter(EmitterContext ctx, List< if (ctx.symbolTable.isStrictOptionEnabled(HINT_UTF8) || ctx.compilerOptions.isUnicodeSource) { - // utf8 source code is true - keep Unicode string as-is - buffers.add(buffer.toString()); + // A byte-backed eval can activate `use utf8` inside its own source. + // Decode only after the parser has applied that lexical pragma; + // whole-source substring detection would misclassify comments and + // quoted text containing the words "use utf8". + buffers.add(ctx.compilerOptions.isByteStringSource + ? decodeUtf8ByteSource(buffer.toString()) + : buffer.toString()); // System.out.println("buffers utf8: " + buffer.toString().length() + " " + buffer.toString()); } else if (ctx.compilerOptions.isEvalbytes) { @@ -317,6 +322,22 @@ public static ParsedString parseRawStringWithDelimiter(EmitterContext ctx, List< return parsed; } + private static String decodeUtf8ByteSource(String source) { + byte[] bytes = new byte[source.length()]; + for (int i = 0; i < source.length(); i++) { + bytes[i] = (byte) (source.charAt(i) & 0xFF); + } + try { + return java.nio.charset.StandardCharsets.UTF_8.newDecoder() + .onMalformedInput(java.nio.charset.CodingErrorAction.REPORT) + .onUnmappableCharacter(java.nio.charset.CodingErrorAction.REPORT) + .decode(java.nio.ByteBuffer.wrap(bytes)) + .toString(); + } catch (java.nio.charset.CharacterCodingException e) { + throw new IllegalArgumentException("Malformed UTF-8 character (fatal)", e); + } + } + public static ParsedString parseRawStrings(Parser parser, EmitterContext ctx, List tokens, int tokenIndex, int stringCount) { return parseRawStrings(parser, ctx, tokens, tokenIndex, stringCount, false); } diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java b/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java index 1fb7426497..c83b85c27f 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeCode.java @@ -27,6 +27,7 @@ import org.perlonjava.runtime.operators.ModuleOperators; import org.perlonjava.runtime.operators.WarnDie; import org.perlonjava.runtime.perlmodule.BHooksEndOfScope; +import org.perlonjava.runtime.perlmodule.Strict; import org.perlonjava.runtime.CoreSubroutineGenerator; import java.lang.invoke.MethodHandle; @@ -2065,7 +2066,15 @@ public static String decodeEvalbytesUtf8Source(String source) { for (int i = 0; i < source.length(); i++) { bytes[i] = (byte) (source.charAt(i) & 0xFF); } - return new String(bytes, StandardCharsets.UTF_8); + try { + return StandardCharsets.UTF_8.newDecoder() + .onMalformedInput(java.nio.charset.CodingErrorAction.REPORT) + .onUnmappableCharacter(java.nio.charset.CodingErrorAction.REPORT) + .decode(java.nio.ByteBuffer.wrap(bytes)) + .toString(); + } catch (java.nio.charset.CharacterCodingException e) { + throw new IllegalArgumentException("Malformed UTF-8 character (fatal)", e); + } } /** @@ -2146,11 +2155,14 @@ public static Class evalStringHelper(RuntimeScalar code, String evalTag, Obje // EXCEPT for evalbytes, which must treat everything as bytes String evalString = code.toString(); boolean evalbytesUtf8Source = ctx.isEvalbytes && shouldDecodeEvalbytesUtf8Source(evalString); - if (evalbytesUtf8Source) { + boolean byteStringUtf8Source = !ctx.isEvalbytes + && code.type == RuntimeScalarType.BYTE_STRING + && (ctx.symbolTable.strictOptionsStack.peek() & Strict.HINT_UTF8) != 0; + if (evalbytesUtf8Source || byteStringUtf8Source) { evalString = decodeEvalbytesUtf8Source(evalString); } boolean hasUnicode = false; - if (evalbytesUtf8Source) { + if (evalbytesUtf8Source || byteStringUtf8Source) { hasUnicode = true; } else if (!ctx.isEvalbytes && code.type != RuntimeScalarType.BYTE_STRING) { for (int i = 0; i < evalString.length(); i++) { @@ -2168,7 +2180,9 @@ public static Class evalStringHelper(RuntimeScalar code, String evalTag, Obje // The eval string can originate from either a Perl STRING or BYTE_STRING scalar. // For BYTE_STRING source we must treat the source as raw bytes (latin-1-ish) and // NOT re-encode characters to UTF-8 when simulating 'non-unicode source'. - boolean isByteStringSource = !ctx.isEvalbytes && code.type == RuntimeScalarType.BYTE_STRING; + boolean isByteStringSource = !ctx.isEvalbytes + && code.type == RuntimeScalarType.BYTE_STRING + && !byteStringUtf8Source; if (hasUnicode) { evalCompilerOptions.isUnicodeSource = true; } @@ -2176,6 +2190,7 @@ public static Class evalStringHelper(RuntimeScalar code, String evalTag, Obje evalCompilerOptions.isEvalbytes = true; } if (isByteStringSource) { + evalCompilerOptions.isUnicodeSource = false; evalCompilerOptions.isByteStringSource = true; } @@ -2197,7 +2212,7 @@ public static Class evalStringHelper(RuntimeScalar code, String evalTag, Obje // Include package name in cache key to ensure source location info is correct per-package int featureFlags = ctx.symbolTable.featureFlagsStack.peek(); String currentPackage = ctx.symbolTable.getCurrentPackage(); - String cacheKey = evalString + '\0' + evalTag + '\0' + hasUnicode + '\0' + ctx.isEvalbytes + '\0' + evalbytesUtf8Source + '\0' + isByteStringSource + '\0' + featureFlags + '\0' + currentPackage; + String cacheKey = evalString + '\0' + evalTag + '\0' + hasUnicode + '\0' + ctx.isEvalbytes + '\0' + evalbytesUtf8Source + '\0' + byteStringUtf8Source + '\0' + isByteStringSource + '\0' + featureFlags + '\0' + currentPackage; Class cachedClass = null; if (!isDebugging) { Map> runtimeEvalCache = evalCache(); @@ -2689,14 +2704,17 @@ public static RuntimeList evalStringWithInterpreter( try { String evalString = code.toString(); boolean evalbytesUtf8Source = ctx.isEvalbytes && shouldDecodeEvalbytesUtf8Source(evalString); - if (evalbytesUtf8Source) { + boolean byteStringUtf8Source = !ctx.isEvalbytes + && code.type == RuntimeScalarType.BYTE_STRING + && (ctx.symbolTable.strictOptionsStack.peek() & Strict.HINT_UTF8) != 0; + if (evalbytesUtf8Source || byteStringUtf8Source) { evalString = decodeEvalbytesUtf8Source(evalString); } evalTrace("evalStringWithInterpreter parse start tag=" + evalTag + " ctx=" + callContext + " fileName=" + ctx.compilerOptions.fileName); // Handle Unicode source detection (same logic as evalStringHelper) boolean hasUnicode = false; - if (evalbytesUtf8Source) { + if (evalbytesUtf8Source || byteStringUtf8Source) { hasUnicode = true; } else if (!ctx.isEvalbytes && code.type != RuntimeScalarType.BYTE_STRING) { for (int i = 0; i < evalString.length(); i++) { @@ -2710,7 +2728,9 @@ public static RuntimeList evalStringWithInterpreter( // Clone compiler options and set isUnicodeSource if needed // Always clone to avoid modifying the original and to set a unique filename CompilerOptions evalCompilerOptions = ctx.compilerOptions.clone(); - boolean isByteStringSource = !ctx.isEvalbytes && code.type == RuntimeScalarType.BYTE_STRING; + boolean isByteStringSource = !ctx.isEvalbytes + && code.type == RuntimeScalarType.BYTE_STRING + && !byteStringUtf8Source; if (hasUnicode) { evalCompilerOptions.isUnicodeSource = true; } @@ -2718,6 +2738,7 @@ public static RuntimeList evalStringWithInterpreter( evalCompilerOptions.isEvalbytes = true; } if (isByteStringSource) { + evalCompilerOptions.isUnicodeSource = false; evalCompilerOptions.isByteStringSource = true; } // Always generate a unique filename for each eval to prevent source location collisions diff --git a/src/test/resources/unit/regex/eval_byte_source_utf8.t b/src/test/resources/unit/regex/eval_byte_source_utf8.t new file mode 100644 index 0000000000..ac5b94a5ab --- /dev/null +++ b/src/test/resources/unit/regex/eval_byte_source_utf8.t @@ -0,0 +1,46 @@ +use strict; +use warnings; +use Test::More tests => 7; + +# Keep this compilation unit Unicode-sourced while individual eval sites +# independently enable or disable the utf8 pragma: é. + +my $quoted_divide = pack('C*', 0x22, 0xC3, 0xB7, 0x22); +ok(!utf8::is_utf8($quoted_divide), 'fixture is an octet string'); + +{ + use utf8; + my $value = eval $quoted_divide; + is($@, '', 'UTF-8 byte source compiles under lexical use utf8'); + is(ord($value), 0xF7, 'lexical use utf8 decodes eval byte source'); +} + +{ + no utf8; + my $value = eval $quoted_divide; + is(ord($value), 0xC3, 'no utf8 retains byte-source interpretation'); +} + +{ + no utf8; + my $source = pack('C*', unpack('C*', "use utf8; \"\x{C3}\x{B7}\"")); + my $value = eval $source; + is(ord($value), 0xF7, 'use utf8 inside byte source decodes following text'); +} + +{ + no utf8; + my @prefix = unpack('C*', 'my $text = "use utf8"; "'); + my $source = pack('C*', @prefix, 0xC3, 0xB7, 0x22); + my $value = eval $source; + is(ord($value), 0xC3, 'quoted use utf8 text does not activate the pragma'); +} + +{ + use utf8; + my $malformed = pack('C*', 0x22, 0xFF, 0x22); + local $SIG{__WARN__} = sub { }; + my $value = eval $malformed; + ok(!defined($value) && $@ =~ /Malformed UTF-8 character/, + 'malformed UTF-8 byte source is fatal'); +} From b2e02ee8d454ac9791cbe82491621dbce0e36f3f Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:13:45 +0200 Subject: [PATCH 16/60] fix(regex): keep Joni syntax errors fatal (#1021) Squashed integration of PR #1021 at d66a5e9f0. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 10 +++++--- .../runtime/regex/RuntimeRegex.java | 25 ++++++++++++++++--- .../unit/regex/joni_syntax_errors_are_fatal.t | 20 +++++++++++++++ 3 files changed, 49 insertions(+), 6 deletions(-) create mode 100644 src/test/resources/unit/regex/joni_syntax_errors_are_fatal.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 58eba7dba9..44f667ae93 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -443,6 +443,10 @@ matcher-specific timeouts on both execution backends. byte semantics and fatal malformed-UTF-8 diagnostics. The focused oracle passes 7/7 on system Perl, JVM, interpreter, and the direct JVM eval compiler. + - [x] Kept Joni syntax/value exceptions fatal for ordinary user-source + compilation while preserving executable-source validation deferral. The + focused oracle passes 7/7 and unchanged forced-Joni `reg_mesg.t` gains 259 + raw passing assertions; 197 parser-acceptance differences remain classified. - [ ] Phase 5: Remove the Java matching backend - [x] Retired the unreachable top-level `(*PRUNE)` text rewrite after native Joni control-verb gates passed under default and forced-Java policy @@ -451,9 +455,9 @@ matcher-specific timeouts on both execution backends. ### Next Steps -1. Finalize PR #1019 after its tracker correction, then integrate the validated - fatal-Joni-syntax commit `d2fd88096` (+259 raw `reg_mesg.t` passes; warning- - free `make`) and publish it as the next stacked slice. +1. Publish the integrated fatal-Joni-syntax slice after combined-stack focused + gates and warning-free `make`, then integrate the validated PRUNE + preprocessor retirement commit `5760874e4` as the next stacked slice. 2. Fix lexical `use bytes` substitution when an upgraded marker regex matches a byte subject, without patching generated fixtures. Regenerate the lossless corpus and prove that chunks 05–10 no longer match literal boundary markers diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java index 94733dcc18..60772dbc15 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java @@ -1,5 +1,6 @@ package org.perlonjava.runtime.regex; +import org.joni.exception.SyntaxException; import org.perlonjava.backend.bytecode.InterpreterState; import org.perlonjava.runtime.WarningBitsRegistry; import org.perlonjava.runtime.operators.PerlUtfString; @@ -501,7 +502,8 @@ private static RuntimeRegex compile(String patternString, String modifiers, int RegexFlags preloadFlags = fromModifiers(modifiers, patternString); UnicodeResolver.preloadUserDefinedProperties( patternString, preloadFlags.isCaseInsensitive()); - return compileSynchronized(patternString, modifiers, lexicalDebugMode, trustedCalloutCount); + return compileSynchronized(patternString, modifiers, lexicalDebugMode, + trustedCalloutCount, false); } /** User properties execute Perl code and therefore cannot be validated while compiling a CV. */ @@ -522,7 +524,8 @@ public static boolean requiresRuntimeUnicodePropertyResolution(String patternStr */ public static void validateLiteralSyntax(String patternString, String modifiers) { try { - compileSynchronized(patternString, stripDebugMarkers(modifiers), debugMode(modifiers), 0); + compileSynchronized(patternString, stripDebugMarkers(modifiers), + debugMode(modifiers), 0, true); } catch (PerlJavaUnimplementedException unsupported) { String message = unsupported.getMessage(); if (message != null && (message.contains("premature end of char-class") @@ -541,7 +544,7 @@ public static void validateLiteralSyntax(String patternString, String modifiers) private static synchronized RuntimeRegex compileSynchronized( String patternString, String modifiers, int lexicalDebugMode, - int trustedCalloutCount) { + int trustedCalloutCount, boolean literalSyntaxValidation) { // Debug logging if (DEBUG_REGEX) { System.err.println("RuntimeRegex.compile: pattern=" + patternString + " modifiers=" + modifiers); @@ -747,6 +750,22 @@ private static synchronized RuntimeRegex compileSynchronized( throw new PerlCompilerException( "Can't find Unicode property definition \"" + invalidProperty + "\""); } + // Joni reports malformed patterns with SyntaxException (including its + // ValueException subclass). These are real compile errors, not missing + // PerlOnJava features, so JPERL_UNIMPLEMENTED=warn must not downgrade them. + boolean validatesExecutableSource = literalSyntaxValidation + && containsExecutableSource(originalPatternString, + regex.regexFlags.isExtended()); + if (e instanceof SyntaxException && !validatesExecutableSource) { + String message = e.getMessage(); + if (literalSyntaxValidation && message != null + && (message.contains("premature end of char-class") + || message.contains("Unclosed character class"))) { + throw new PerlCompilerException("Unmatched [ in regex m/" + + originalPatternString + "/"); + } + throw new PerlCompilerException(message); + } // PerlJavaUnimplementedException extends PerlCompilerException, so check // the more specific type first. Real syntax errors (PerlCompilerException // but NOT PerlJavaUnimplementedException) are always fatal. diff --git a/src/test/resources/unit/regex/joni_syntax_errors_are_fatal.t b/src/test/resources/unit/regex/joni_syntax_errors_are_fatal.t new file mode 100644 index 0000000000..de3f9856f8 --- /dev/null +++ b/src/test/resources/unit/regex/joni_syntax_errors_are_fatal.t @@ -0,0 +1,20 @@ +use strict; +use warnings; +use Test::More tests => 7; + +local $ENV{JPERL_UNIMPLEMENTED} = 'warn'; +my @warnings; +local $SIG{__WARN__} = sub { push @warnings, @_ }; + +for my $case ( + [ 'qr/(?;x/', 'invalid group introducer' ], + [ 'qr/[z-a]/', 'invalid character range' ], + [ 'qr/x{2147483648}/', 'oversized quantifier' ], +) { + my ($source, $description) = @$case; + my $ok = eval "$source; 1"; + ok(!$ok, "$description is rejected"); + ok(length($@), "$description populates the compile error"); +} + +is(scalar @warnings, 0, 'syntax errors are not downgraded to warnings'); From e517492cbbb052756d2439733a5185ffb80c5f86 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:13:55 +0200 Subject: [PATCH 17/60] fix(regex): reject invalid Perl inline regex options (#1022) Squashed integration of PR #1022 at 9cca1c0f1. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 13 +++++-- .../unit/regex/joni_inline_option_errors.t | 24 +++++++++++++ third_party/joni/src/org/joni/Parser.java | 36 +++++++++++++++++++ .../src/org/joni/exception/ErrorMessages.java | 14 ++++++++ .../joni/test/org/joni/test/TestPerl.java | 8 +++++ 5 files changed, 92 insertions(+), 3 deletions(-) create mode 100644 src/test/resources/unit/regex/joni_inline_option_errors.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 44f667ae93..0141601794 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -424,6 +424,9 @@ matcher-specific timeouts on both execution backends. - [x] Normalized explicit `Is_*` property/value assignments and the colon delimiter (PR #1019), gaining exactly 44,944 generated assertions on both execution backends without changing any plan. + - [x] Rejected 40 invalid Perl inline option/group-name forms in forked Joni + with exact JVM/interpreter `reg_mesg.t` parity, reducing residual Joni-only + acceptance differences from 198 to 158 (`028602adc`). - [ ] Fix byte-mode substitution of upgraded marker regexes so chunks 05–10 exercise real boundary subjects, then close the property and boundary failures with exact JVM/interpreter plan and semantic parity before marking @@ -451,13 +454,17 @@ matcher-specific timeouts on both execution backends. - [x] Retired the unreachable top-level `(*PRUNE)` text rewrite after native Joni control-verb gates passed under default and forced-Java policy (`5760874e4`; 316 preprocessor lines removed). + - [x] Removed the disabled invalid-brace pass and its exclusive helpers + (`4be6a48e3`; 208 preprocessor lines removed), retaining active Perl/Joni + quantifier diagnostics as explicit focused hard/TODO gates. - [ ] Phase 6: Integration and release ### Next Steps -1. Publish the integrated fatal-Joni-syntax slice after combined-stack focused - gates and warning-free `make`, then integrate the validated PRUNE - preprocessor retirement commit `5760874e4` as the next stacked slice. +1. Publish the integrated inline-option parser slice after combined-stack + focused gates and warning-free `make`, then integrate the validated PRUNE + and dead invalid-brace preprocessor retirement commits `5760874e4` and + `4be6a48e3` as the next stacked slice. 2. Fix lexical `use bytes` substitution when an upgraded marker regex matches a byte subject, without patching generated fixtures. Regenerate the lossless corpus and prove that chunks 05–10 no longer match literal boundary markers diff --git a/src/test/resources/unit/regex/joni_inline_option_errors.t b/src/test/resources/unit/regex/joni_inline_option_errors.t new file mode 100644 index 0000000000..5b4b465b23 --- /dev/null +++ b/src/test/resources/unit/regex/joni_inline_option_errors.t @@ -0,0 +1,24 @@ +use strict; +use warnings; +use Test::More tests => 15; + +local $ENV{JPERL_UNIMPLEMENTED} = 'warn'; +my @warnings; +local $SIG{__WARN__} = sub { push @warnings, @_ }; + +for my $case ( + [ 'qr/(?<;name>match)/', 'Group name must start with a non-digit word character' ], + [ 'qr/(?^-i:foo)/', 'Sequence (?^-...) not recognized' ], + [ 'qr/(?^d:foo)/', 'Sequence (?^d...) not recognized' ], + [ 'qr/(?^lu:foo)/', 'Regexp modifiers "l" and "u" are mutually exclusive' ], + [ 'qr/(?da:foo)/', 'Regexp modifiers "d" and "a" are mutually exclusive' ], + [ 'qr/(?lil:foo)/', 'Regexp modifier "l" may not appear twice' ], + [ 'qr/(?aaia:foo)/', 'Regexp modifier "a" may appear a maximum of twice' ], +) { + my ($source, $expected) = @$case; + my $ok = eval "$source; 1"; + ok(!$ok, "$source is rejected"); + like($@, qr/^\Q$expected\E/, "$source reports the Perl diagnostic"); +} + +is(scalar @warnings, 0, 'fatal option diagnostics do not warn'); diff --git a/third_party/joni/src/org/joni/Parser.java b/third_party/joni/src/org/joni/Parser.java index 68177885c4..150b01caac 100644 --- a/third_party/joni/src/org/joni/Parser.java +++ b/third_party/joni/src/org/joni/Parser.java @@ -625,6 +625,12 @@ private Node parseEnclose(TokenType term) { option = bsOnOff(option, Option.CAPTURE_GROUP, false); option = bsOnOff(option, Option.PERL_ASCII_STRICT, true); fetch(); + if (c == '-') { + newSyntaxException(PERL_CARET_MINUS_OPTION_NOT_RECOGNIZED); + } + if (c == 'd') { + newSyntaxException(PERL_CARET_D_OPTION_NOT_RECOGNIZED); + } } else { newSyntaxException(UNDEFINED_GROUP_OPTION); } @@ -642,6 +648,9 @@ private Node parseEnclose(TokenType term) { case 'u': boolean neg = false; int asciiModifierCount = 0; + boolean sawDefaultCharset = false; + boolean sawLocaleCharset = false; + boolean sawUnicodeCharset = false; while (true) { switch(c) { case ':': @@ -691,7 +700,13 @@ private Node parseEnclose(TokenType term) { case 'a': /* limits \d, \s, \w and POSIX brackets to ASCII range */ if ((syntax.op2OptionPerl() || syntax.op2OptionRuby()) && !neg) { + if (syntax.op2OptionPerl() && sawDefaultCharset) { + newSyntaxException(PERL_MODIFIERS_D_AND_A_MUTUALLY_EXCLUSIVE); + } asciiModifierCount++; + if (syntax.op2OptionPerl() && asciiModifierCount > 2) { + newSyntaxException(PERL_MODIFIER_A_MAXIMUM_TWICE); + } option = bsOnOff(option, Option.ASCII_RANGE, false); option = bsOnOff(option, Option.POSIX_BRACKET_ALL_RANGE, true); option = bsOnOff(option, Option.WORD_BOUND_ALL_RANGE, true); @@ -704,6 +719,10 @@ private Node parseEnclose(TokenType term) { } case 'u': if ((syntax.op2OptionPerl() || syntax.op2OptionRuby()) && !neg) { + if (syntax.op2OptionPerl() && sawLocaleCharset) { + newSyntaxException(PERL_MODIFIERS_L_AND_U_MUTUALLY_EXCLUSIVE); + } + sawUnicodeCharset = true; option = bsOnOff(option, Option.ASCII_RANGE, true); option = bsOnOff(option, Option.POSIX_BRACKET_ALL_RANGE, true); option = bsOnOff(option, Option.WORD_BOUND_ALL_RANGE, true); @@ -715,6 +734,10 @@ private Node parseEnclose(TokenType term) { case 'd': if (syntax.op2OptionPerl() && !neg) { + if (asciiModifierCount > 0) { + newSyntaxException(PERL_MODIFIERS_D_AND_A_MUTUALLY_EXCLUSIVE); + } + sawDefaultCharset = true; option = bsOnOff(option, Option.ASCII_RANGE, true); option = bsOnOff(option, Option.PERL_ASCII_STRICT, true); } else if (syntax.op2OptionRuby() && !neg) { @@ -728,6 +751,13 @@ private Node parseEnclose(TokenType term) { case 'l': if (syntax.op2OptionPerl() && !neg) { + if (sawUnicodeCharset) { + newSyntaxException(PERL_MODIFIERS_L_AND_U_MUTUALLY_EXCLUSIVE); + } + if (sawLocaleCharset) { + newSyntaxException(PERL_MODIFIER_L_MAY_NOT_APPEAR_TWICE); + } + sawLocaleCharset = true; option = bsOnOff(option, Option.ASCII_RANGE, true); option = bsOnOff(option, Option.PERL_ASCII_STRICT, true); } else { @@ -905,6 +935,12 @@ private boolean startsWith(String value) { } private Node parseEncloseNamedGroup2(boolean listCapture) { + if (syntax.op2OptionPerl() && left()) { + int first = enc.mbcToCode(bytes, p, stop); + if (enc.isDigit(first) || !enc.isWord(first)) { + newValueException(PERL_GROUP_NAME_MUST_START_WITH_WORD); + } + } int nm = p; fetchName(c, false); int nameEnd = value; diff --git a/third_party/joni/src/org/joni/exception/ErrorMessages.java b/third_party/joni/src/org/joni/exception/ErrorMessages.java index b80340b720..d36a50c32b 100644 --- a/third_party/joni/src/org/joni/exception/ErrorMessages.java +++ b/third_party/joni/src/org/joni/exception/ErrorMessages.java @@ -57,6 +57,20 @@ public interface ErrorMessages extends org.jcodings.exception.ErrorMessages { String INVALID_LOOK_BEHIND_PATTERN = "invalid pattern in look-behind"; String INVALID_REPEAT_RANGE_PATTERN = "invalid repeat range {lower,upper}"; String INVALID_CONDITION_PATTERN = "invalid conditional pattern"; + String PERL_GROUP_NAME_MUST_START_WITH_WORD = + "Group name must start with a non-digit word character"; + String PERL_CARET_MINUS_OPTION_NOT_RECOGNIZED = + "Sequence (?^-...) not recognized"; + String PERL_CARET_D_OPTION_NOT_RECOGNIZED = + "Sequence (?^d...) not recognized"; + String PERL_MODIFIERS_L_AND_U_MUTUALLY_EXCLUSIVE = + "Regexp modifiers \"l\" and \"u\" are mutually exclusive"; + String PERL_MODIFIERS_D_AND_A_MUTUALLY_EXCLUSIVE = + "Regexp modifiers \"d\" and \"a\" are mutually exclusive"; + String PERL_MODIFIER_L_MAY_NOT_APPEAR_TWICE = + "Regexp modifier \"l\" may not appear twice"; + String PERL_MODIFIER_A_MAXIMUM_TWICE = + "Regexp modifier \"a\" may appear a maximum of twice"; /* values error (syntax error) */ String TOO_BIG_NUMBER = "too big number"; diff --git a/third_party/joni/test/org/joni/test/TestPerl.java b/third_party/joni/test/org/joni/test/TestPerl.java index dd27926c11..a4fb6f6bf6 100755 --- a/third_party/joni/test/org/joni/test/TestPerl.java +++ b/third_party/joni/test/org/joni/test/TestPerl.java @@ -21,6 +21,7 @@ import org.joni.Option; import org.joni.Syntax; +import org.joni.exception.ErrorMessages; import org.jcodings.Encoding; import org.jcodings.specific.ASCIIEncoding; @@ -43,6 +44,13 @@ public Syntax syntax() { } @Override public void test() throws Exception { + xerrs("(?<;name>match)", ErrorMessages.PERL_GROUP_NAME_MUST_START_WITH_WORD); + xerrs("(?^-i:foo)", ErrorMessages.PERL_CARET_MINUS_OPTION_NOT_RECOGNIZED); + xerrs("(?^d:foo)", ErrorMessages.PERL_CARET_D_OPTION_NOT_RECOGNIZED); + xerrs("(?^lu:foo)", ErrorMessages.PERL_MODIFIERS_L_AND_U_MUTUALLY_EXCLUSIVE); + xerrs("(?da:foo)", ErrorMessages.PERL_MODIFIERS_D_AND_A_MUTUALLY_EXCLUSIVE); + xerrs("(?lil:foo)", ErrorMessages.PERL_MODIFIER_L_MAY_NOT_APPEAR_TWICE); + xerrs("(?aaia:foo)", ErrorMessages.PERL_MODIFIER_A_MAXIMUM_TWICE); } @org.junit.Test(timeout = 5000) From f544726dc644175115f0cf48010668daf420d064 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:13:56 +0200 Subject: [PATCH 18/60] refactor(regex): retire obsolete preprocessor passes (#1023) Squashed integration of PR #1023 at 5bba948ee. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 8 +- .../runtime/regex/RegexPreprocessor.java | 524 ------------------ .../regex/prune_preprocessor_retirement.t | 15 + .../unit/regex/quantifier_brace_semantics.t | 37 ++ 4 files changed, 56 insertions(+), 528 deletions(-) create mode 100644 src/test/resources/unit/regex/prune_preprocessor_retirement.t create mode 100644 src/test/resources/unit/regex/quantifier_brace_semantics.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 0141601794..42edf4c89c 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -461,10 +461,10 @@ matcher-specific timeouts on both execution backends. ### Next Steps -1. Publish the integrated inline-option parser slice after combined-stack - focused gates and warning-free `make`, then integrate the validated PRUNE - and dead invalid-brace preprocessor retirement commits `5760874e4` and - `4be6a48e3` as the next stacked slice. +1. Publish the integrated PRUNE and dead invalid-brace preprocessor retirement + slice after its warning-free combined `make`, then integrate the validated + forked-Joni braced-octal diagnostics/tokenization slice. Continue retiring + independently proven Java-only preprocessor rewrites as they complete. 2. Fix lexical `use bytes` substitution when an upgraded marker regex matches a byte subject, without patching generated fixtures. Regenerate the lossless corpus and prove that chunks 05–10 no longer match literal boundary markers diff --git a/src/main/java/org/perlonjava/runtime/regex/RegexPreprocessor.java b/src/main/java/org/perlonjava/runtime/regex/RegexPreprocessor.java index 2c18e803c2..5997a0189c 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RegexPreprocessor.java +++ b/src/main/java/org/perlonjava/runtime/regex/RegexPreprocessor.java @@ -150,11 +150,6 @@ private static String preProcessRegexInternal(String s, RegexFlags regexFlags) { duplicateNameCounter = 0; warningsOnUse.clear(); - // First, escape invalid quantifier braces (Perl compatibility) - // DISABLED: Causes test regressions - needs more work - // s = escapeInvalidQuantifierBraces(s); - - s = translateTopLevelPruneVerbs(s); s = convertPythonStyleGroups(s); s = transformSimpleConditionals(s); s = removeUnderscoresFromEscapes(s); @@ -174,321 +169,6 @@ private static String preProcessRegexInternal(String s, RegexFlags regexFlags) { return preferOmniHolderLiteralAlternation(result); } - /** - * Translate the common top-level forms of Perl's {@code (*PRUNE)} cut - * verb into constructs supported by {@link java.util.regex.Pattern}. - * - *

Within one alternation branch, PRUNE makes everything already - * consumed atomic. If that branch later fails, alternatives to its right - * must also be rejected when the pruned prefix matched at the same start - * position. The latter can be represented by a negative lookahead when - * the prefix contains no groups (and therefore cannot perturb capture - * numbering when copied into the guard):

- * - * 
A(*PRUNE)B|C  ->  (?>A)B|(?!A)C
- * - *

Nested PRUNE sites and grouped prefixes followed by alternatives - * remain on the existing explicit-unimplemented path. Keeping this - * translation deliberately structural avoids silently weakening cut - * semantics for patterns that need a full Perl backtracking VM.

- */ - private static String translateTopLevelPruneVerbs(String pattern) { - final String prune = "(*PRUNE)"; - pattern = translatePruneVerbsInNestedGroups(pattern); - int searchFrom = 0; - - while (true) { - int pruneAt = findTopLevelToken(pattern, prune, searchFrom); - if (pruneAt < 0) return pattern; - - int branchStart = findPreviousTopLevelPipe(pattern, pruneAt) + 1; - String prefix = pattern.substring(branchStart, pruneAt); - List laterPipes = findFollowingTopLevelPipes(pattern, pruneAt + prune.length()); - - if (!laterPipes.isEmpty() && !isPruneGuardSafe(prefix)) { - // Leave the verb untouched so handleParentheses reports the - // established, honest unsupported-feature diagnostic. - searchFrom = pruneAt + prune.length(); - continue; - } - - String atomicPrefix = "(?>" + prefix + ")"; - String guard = laterPipes.isEmpty() ? "" : "(?!" + prefix + ")"; - StringBuilder rewritten = new StringBuilder( - pattern.length() + atomicPrefix.length() + guard.length() * laterPipes.size()); - - rewritten.append(pattern, 0, branchStart).append(atomicPrefix); - int cursor = pruneAt + prune.length(); - for (int pipeAt : laterPipes) { - rewritten.append(pattern, cursor, pipeAt + 1).append(guard); - cursor = pipeAt + 1; - } - rewritten.append(pattern, cursor, pattern.length()); - pattern = rewritten.toString(); - - // Continue after the atomic prefix. Any later PRUNE in this same - // branch may atomize the already-translated prefix again, which - // is both valid and equivalent to the successive Perl cuts. - searchFrom = branchStart + atomicPrefix.length(); - } - } - - private static String translatePruneVerbsInNestedGroups(String pattern) { - if (!pattern.contains("(*PRUNE)")) return pattern; - - StringBuilder rewritten = new StringBuilder(pattern.length()); - boolean escaped = false; - boolean inClass = false; - boolean quoted = false; - for (int i = 0; i < pattern.length(); i++) { - char ch = pattern.charAt(i); - if (escaped) { - rewritten.append(ch); - if (!inClass && ch == 'Q') quoted = true; - else if (!inClass && ch == 'E') quoted = false; - escaped = false; - continue; - } - if (ch == '\\') { - rewritten.append(ch); - escaped = true; - continue; - } - if (quoted) { - rewritten.append(ch); - continue; - } - if (ch == '[' && !inClass) { - inClass = true; - rewritten.append(ch); - continue; - } - if (ch == ']' && inClass) { - inClass = false; - rewritten.append(ch); - continue; - } - if (inClass || ch != '(' || pattern.startsWith("(*", i)) { - rewritten.append(ch); - continue; - } - - int close = findRegexGroupEnd(pattern, i); - int contentStart = pruneGroupContentStart(pattern, i, close); - if (close < 0 || contentStart < 0 - || !pattern.substring(contentStart, close).contains("(*PRUNE)")) { - rewritten.append(ch); - continue; - } - - rewritten.append(pattern, i, contentStart); - rewritten.append(translateTopLevelPruneVerbs( - pattern.substring(contentStart, close))); - rewritten.append(')'); - i = close; - } - return rewritten.toString(); - } - - private static int pruneGroupContentStart(String pattern, int open, int close) { - if (close < 0 || open + 1 >= close || pattern.charAt(open + 1) != '?') { - return open + 1; - } - if (open + 2 >= close) return -1; - char kind = pattern.charAt(open + 2); - if (kind == ':' || kind == '=' || kind == '!' || kind == '>') return open + 3; - if (kind == '#' || kind == '(' || kind == '{') return -1; - if (kind == '?' && open + 3 < close && pattern.charAt(open + 3) == '{') return -1; - if (kind == '<') { - if (open + 3 < close - && (pattern.charAt(open + 3) == '=' || pattern.charAt(open + 3) == '!')) { - return open + 4; - } - int end = pattern.indexOf('>', open + 3); - return end >= 0 && end < close ? end + 1 : -1; - } - if (kind == '\'') { - int end = pattern.indexOf('\'', open + 3); - return end >= 0 && end < close ? end + 1 : -1; - } - if (kind == 'P' && open + 3 < close && pattern.charAt(open + 3) == '<') { - int end = pattern.indexOf('>', open + 4); - return end >= 0 && end < close ? end + 1 : -1; - } - - // Inline modifier groups, including Perl's normalized (?^x:...), - // have their regex body after the first colon. - int colon = pattern.indexOf(':', open + 2); - return colon >= 0 && colon < close ? colon + 1 : -1; - } - - private static int findRegexGroupEnd(String pattern, int open) { - int depth = 1; - boolean escaped = false; - boolean inClass = false; - boolean quoted = false; - for (int i = open + 1; i < pattern.length(); i++) { - char ch = pattern.charAt(i); - if (escaped) { - if (!inClass && ch == 'Q') quoted = true; - else if (!inClass && ch == 'E') quoted = false; - escaped = false; - continue; - } - if (ch == '\\') { - escaped = true; - continue; - } - if (quoted) continue; - if (ch == '[' && !inClass) { - inClass = true; - continue; - } - if (ch == ']' && inClass) { - inClass = false; - continue; - } - if (inClass) continue; - if (ch == '(') depth++; - else if (ch == ')' && --depth == 0) return i; - } - return -1; - } - - private static int findTopLevelToken(String pattern, String token, int start) { - int depth = 0; - boolean escaped = false; - boolean inClass = false; - boolean quoted = false; - for (int i = Math.max(0, start); i < pattern.length(); i++) { - char ch = pattern.charAt(i); - if (escaped) { - if (!inClass && ch == 'Q') quoted = true; - else if (!inClass && ch == 'E') quoted = false; - escaped = false; - continue; - } - if (ch == '\\') { - escaped = true; - continue; - } - if (quoted) continue; - if (ch == '[' && !inClass) { - inClass = true; - continue; - } - if (ch == ']' && inClass) { - inClass = false; - continue; - } - if (inClass) continue; - if (depth == 0 && pattern.startsWith(token, i)) return i; - if (ch == '(') depth++; - else if (ch == ')' && depth > 0) depth--; - } - return -1; - } - - private static int findPreviousTopLevelPipe(String pattern, int end) { - int depth = 0; - int lastPipe = -1; - boolean escaped = false; - boolean inClass = false; - boolean quoted = false; - for (int i = 0; i < end; i++) { - char ch = pattern.charAt(i); - if (escaped) { - if (!inClass && ch == 'Q') quoted = true; - else if (!inClass && ch == 'E') quoted = false; - escaped = false; - continue; - } - if (ch == '\\') { - escaped = true; - continue; - } - if (quoted) continue; - if (ch == '[' && !inClass) { - inClass = true; - continue; - } - if (ch == ']' && inClass) { - inClass = false; - continue; - } - if (inClass) continue; - if (ch == '(') depth++; - else if (ch == ')' && depth > 0) depth--; - else if (ch == '|' && depth == 0) lastPipe = i; - } - return lastPipe; - } - - private static List findFollowingTopLevelPipes(String pattern, int start) { - List pipes = new ArrayList<>(); - int depth = 0; - boolean escaped = false; - boolean inClass = false; - boolean quoted = false; - for (int i = 0; i < pattern.length(); i++) { - char ch = pattern.charAt(i); - if (escaped) { - if (!inClass && ch == 'Q') quoted = true; - else if (!inClass && ch == 'E') quoted = false; - escaped = false; - continue; - } - if (ch == '\\') { - escaped = true; - continue; - } - if (quoted) continue; - if (ch == '[' && !inClass) { - inClass = true; - continue; - } - if (ch == ']' && inClass) { - inClass = false; - continue; - } - if (inClass) continue; - if (ch == '(') depth++; - else if (ch == ')' && depth > 0) depth--; - else if (i >= start && ch == '|' && depth == 0) pipes.add(i); - } - return pipes; - } - - private static boolean isPruneGuardSafe(String prefix) { - boolean escaped = false; - boolean inClass = false; - boolean quoted = false; - for (int i = 0; i < prefix.length(); i++) { - char ch = prefix.charAt(i); - if (escaped) { - if (!inClass && ch == 'Q') quoted = true; - else if (!inClass && ch == 'E') quoted = false; - escaped = false; - continue; - } - if (ch == '\\') { - escaped = true; - continue; - } - if (quoted) continue; - if (ch == '[' && !inClass) { - inClass = true; - continue; - } - if (ch == ']' && inClass) { - inClass = false; - continue; - } - if (!inClass && (ch == '(' || ch == ')')) return false; - } - return true; - } - private static String optimizeTerminatedWhitespaceQuantifiers(String pattern) { StringBuilder result = new StringBuilder(pattern.length()); int i = 0; @@ -906,210 +586,6 @@ private static boolean looksLikeQuotedStarAlternate(String leftAlt) { return leftAlt.contains("(?:'[^") && leftAlt.contains("|\"[^"); } - /** - * Escape unescaped braces that don't form valid quantifiers. - * Perl allows invalid quantifier braces and treats them as literals. - * Java Pattern.compile() rejects them, so we must escape them. - *

- * Valid quantifiers: {n}, {n,}, {n,m} where n and m are non-negative integers - * Invalid quantifiers: {(.*?)}, {abc}, {}, {,5}, etc. - *

- * IMPORTANT: This is a high-risk preprocessing step that modifies brace characters. - * Known edge cases that must be handled correctly: - *

- * 1. ESCAPE SEQUENCES WITH BRACES (must NOT be escaped): - * - \N{name} - Named Unicode character (e.g., \N{LATIN SMALL LETTER A}) - * - \x{...} - Hexadecimal character code (e.g., \x{1F600}) - * - \o{...} - Octal character code (e.g., \o{777}) - * - \p{...} - Unicode property (e.g., \p{Letter}) - * - \P{...} - Negated Unicode property (e.g., \P{Number}) - * - \g{...} - Named or relative backreference (e.g., \g{name}, \g{-1}) - * Currently handled: N, x, o, p, P, g - *

- * 2. CHARACTER CLASSES (braces inside [...] are always literal): - * - [a{3}] means "match 'a', '{', '3', or '}'" not "match 'aaa'" - * - Nested classes like [a-z[0-9]{3}] must track nesting depth - *

- * 3. VALID QUANTIFIERS (must NOT be escaped): - * - {n} - exactly n times (e.g., a{3}) - * - {n,} - n or more times (e.g., a{2,}) - * - {n,m} - between n and m times (e.g., a{2,5}) - *

- * 4. ALREADY ESCAPED BRACES (must NOT be double-escaped): - * - \{ and \} should remain as-is - * - Track backslash escaping carefully to avoid double-escaping - *

- * 5. POSSESSIVE AND LAZY QUANTIFIERS: - * - {n}+ (possessive) and {n}? (lazy) should work with valid quantifiers - *

- * POTENTIAL ISSUES NOT YET HANDLED: - * - Extended bracketed character classes: (?[...]) may contain braces - * - Conditional patterns: (?(condition){yes}{no}) uses braces for branches - * - Subroutine definitions: (?(DEFINE)(?...)) may have complex nesting - * - Code blocks: (?{...}) and (??{...}) use braces but are handled elsewhere - * - Named capture definitions: (?...) - are braces allowed in names? - * - Unicode named sequences: \N{...} may contain nested braces in some contexts - *

- * If new regex features are added that use braces, this function MUST be updated. - * Test changes thoroughly with unit/regex/unescaped_braces.t and regex test suite. - */ - private static String escapeInvalidQuantifierBraces(String pattern) { - StringBuilder result = new StringBuilder(); - boolean inCharClass = false; - boolean escaped = false; - - for (int i = 0; i < pattern.length(); i++) { - char c = pattern.charAt(i); - - // Handle escape sequences - if (escaped) { - result.append(c); - - // Check if this is an escape sequence that uses braces: \N{...}, \x{...}, \o{...}, \p{...}, \P{...}, \g{...} - if ((c == 'N' || c == 'x' || c == 'o' || c == 'p' || c == 'P' || c == 'g') && - i + 1 < pattern.length() && pattern.charAt(i + 1) == '{') { - // Skip the entire escape sequence with braces - result.append('{'); - i++; // Move past '{' - int braceDepth = 1; - i++; // Move to first character inside braces - - while (i < pattern.length() && braceDepth > 0) { - char ch = pattern.charAt(i); - result.append(ch); - if (ch == '\\' && i + 1 < pattern.length()) { - // Skip escaped character inside the escape sequence - i++; - if (i < pattern.length()) { - result.append(pattern.charAt(i)); - } - } else if (ch == '{') { - braceDepth++; - } else if (ch == '}') { - braceDepth--; - } - i++; - } - i--; // Back up one since the loop will increment - } - - escaped = false; - continue; - } - - if (c == '\\') { - result.append(c); - escaped = true; - continue; - } - - // Track character class boundaries (braces inside [...] are always literal) - if (c == '[') { - inCharClass = true; - result.append(c); - continue; - } - if (c == ']') { - inCharClass = false; - result.append(c); - continue; - } - - // Only process braces outside character classes - if (!inCharClass && c == '{') { - // Look ahead to check if this is a valid quantifier - int closePos = findMatchingCloseBraceForEscape(pattern, i); - if (closePos > 0 && isValidQuantifierContent(pattern, i + 1, closePos)) { - result.append(c); // Keep valid quantifier as-is - } else { - result.append("\\{"); // Escape invalid quantifier - } - } else if (!inCharClass && c == '}') { - // Check if this closes a quantifier that we kept unescaped - if (!closesValidQuantifier(result, pattern, i)) { - result.append("\\}"); // Escape unmatched closing brace - } else { - result.append(c); - } - } else { - result.append(c); - } - } - - return result.toString(); - } - - /** - * Find the position of closing brace that matches opening brace at pos. - * Returns -1 if no matching brace found. - */ - private static int findMatchingCloseBraceForEscape(String pattern, int openPos) { - for (int i = openPos + 1; i < pattern.length(); i++) { - char c = pattern.charAt(i); - if (c == '\\') { - i++; // Skip escaped character - continue; - } - if (c == '}') { - return i; - } - } - return -1; // No closing brace found - } - - /** - * Check if content between braces forms a valid quantifier. - * Valid: {n}, {n,}, {n,m} where n and m are non-negative integers - * Invalid: {(.*?)}, {abc}, {}, {,5}, etc. - */ - private static boolean isValidQuantifierContent(String pattern, int start, int end) { - if (start >= end) { - return false; // Empty braces {} - } - - String content = pattern.substring(start, end); - - // Check for {n}, {n,}, or {n,m} pattern - if (content.matches("\\d+")) { - return true; // {n} - } - if (content.matches("\\d+,")) { - return true; // {n,} - } - return content.matches("\\d+,\\d+"); // {n,m} - } - - /** - * Check if closing brace at position closePos closes a valid quantifier - * that we kept unescaped in the result buffer. - */ - private static boolean closesValidQuantifier(StringBuilder result, String pattern, int closePos) { - // Find the most recent unescaped opening brace in result - int openPos = -1; - for (int i = result.length() - 1; i >= 0; i--) { - if (result.charAt(i) == '{') { - // Check if it's escaped - int backslashCount = 0; - for (int j = i - 1; j >= 0 && result.charAt(j) == '\\'; j--) { - backslashCount++; - } - if (backslashCount % 2 == 0) { - // Even number of backslashes (or zero) means { is not escaped - openPos = i; - break; - } - } - } - - if (openPos < 0) { - return false; // No unescaped opening brace found - } - - // Extract content and validate - String content = result.substring(openPos + 1); - return content.matches("\\d+") || content.matches("\\d+,") || content.matches("\\d+,\\d+"); - } - /** * Expand characters with multi-character case folds into alternations. * For example: ß → (?:ß|ss|SS|Ss|sS) diff --git a/src/test/resources/unit/regex/prune_preprocessor_retirement.t b/src/test/resources/unit/regex/prune_preprocessor_retirement.t new file mode 100644 index 0000000000..027b60e270 --- /dev/null +++ b/src/test/resources/unit/regex/prune_preprocessor_retirement.t @@ -0,0 +1,15 @@ +use strict; +use warnings; +use Test::More; + +ok('AB' =~ /A(*PRUNE)B|AC/, 'straight PRUNE path can succeed'); +ok('AC' !~ /A(*PRUNE)B|AC/, 'PRUNE cuts later alternative at same start'); +ok('xxAC' !~ /A(*PRUNE)B|AC/, 'each later search start keeps its own cut'); +ok('xca' =~ /a(*PRUNE)b|c/, 'search may succeed at an earlier alternative start'); +ok('aa' !~ /a+(*PRUNE)a/, 'PRUNE discards greedy quantifier backtracking'); +is(do { 'AAABC' =~ /A+?(*PRUNE)BC/; $& }, 'ABC', + 'lazy quantifier retries only at a later start after PRUNE'); +ok('ac' !~ /(?:(?=a)a(*PRUNE)b|ac)/, 'nested noncapture PRUNE cuts its group'); +ok('zab' =~ /z(?:a(*PRUNE)b|ac)/, 'nested PRUNE success preserves outer match'); + +done_testing; diff --git a/src/test/resources/unit/regex/quantifier_brace_semantics.t b/src/test/resources/unit/regex/quantifier_brace_semantics.t new file mode 100644 index 0000000000..faac1eeff4 --- /dev/null +++ b/src/test/resources/unit/regex/quantifier_brace_semantics.t @@ -0,0 +1,37 @@ +use strict; +use warnings; +use Test::More; + +{ + no warnings 'regexp'; + ok('a{b' =~ /\Aa{b\z/, + 'an unclosed non-quantifier opening brace is literal'); + ok('a{word}b' =~ /\Aa{word}b\z/, + 'non-numeric braces after an atom are literal'); + ok('a{1,2,3}' =~ /\Aa{1,2,3}\z/, + 'too many comma fields make braces literal'); + ok('a}b' =~ /\Aa}b\z/, 'an unmatched closing brace is literal'); +} + +ok('aa' =~ /\Aa{2}\z/, 'exact quantifier remains valid'); +ok('aaa' =~ /\Aa{2,3}\z/, 'bounded quantifier remains valid'); +ok('aa' =~ /\Aa{,2}\z/, 'omitted-minimum quantifier remains valid'); + +sub compile_error { + my ($source) = @_; + local $SIG{__WARN__} = sub {}; + eval "qr/$source/"; + return $@; +} + +TODO: { + local $TODO = 'Joni quantifier diagnostic parity'; + like(compile_error('a{2}{3}'), qr/^Nested quantifiers in regex/, + 'adjacent brace quantifiers are rejected'); + like(compile_error('a{2}?+'), qr/^Nested quantifiers in regex/, + 'conflicting lazy and possessive suffixes are rejected'); + like(compile_error('a{01}'), qr/^Invalid quantifier in \{,\} in regex/, + 'a leading-zero count is rejected'); +} + +done_testing; From bf3eb616390fe1052c5aeda69fa9cbed63c17f78 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:13:56 +0200 Subject: [PATCH 19/60] fix(regex): preserve Joni byte-mode pattern semantics (#1024) Squashed integration of PR #1024 at f22de2d58. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 87 ++++++++++--------- docs/reference/feature-matrix.md | 8 +- .../runtime/regex/JoniRegexPattern.java | 44 ++++++++-- .../runtime/regex/RuntimeRegex.java | 39 ++++++++- .../bytes_upgraded_pattern_substitution.t | 43 +++++++++ 5 files changed, 165 insertions(+), 56 deletions(-) create mode 100644 src/test/resources/unit/regex/bytes_upgraded_pattern_substitution.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 42edf4c89c..2316220f3c 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,35 +224,30 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The combined integration stack now includes immutable Joni offset-map reuse, -regex-set property preservation, bounded catastrophic-backtracking safeguards, -user-property diagnostics and provenance, and focused built-in Unicode alias -normalization. The focused property gates pass 39/39, 12/12, 8/8, and 16/16; -unchanged `regexp_unicode_prop.t` passes 1,110/1,110 on JVM and interpreter. -Draft PR #1016 preserves each validated prerequisite head and passed a -warning-free full `make` on its corrected integration history. - -The lossless generated `TestProp.pl` fixture now exposes ten previously gated -`uniprops*.t` files. The exact-history JVM run executes all 290,912 planned -assertions with zero timeouts or incomplete files, at 175,768 passing and -115,144 failing. PR 958 recorded zero plan for all ten files, so this is new -coverage rather than a regression, but it reopens Phase 3's full-corpus exit -gate. The interpreter produces the exact same 175,768/290,912 split, with -identical per-file plans and no timeout or incomplete file. Semantic-key -validation found that all 123,411 generated records -in chunks 05–10 retain literal UTF-8 boundary markers in both subjects and -patterns. Their 122,620 apparent passing assertions are therefore vacuous, -not evidence of boundary or folding parity. Chunks 01–04 contain the genuine -property and hand-written boundary evidence. An exact upstream-stage reducer -shows that lexical `use bytes` fails to replace the upgraded marker regex in a -byte subject (system Perl 4/4, JVM/interpreter 2/4); that byte-mode substitution -gap must be fixed before chunks 05–10 can become an authoritative boundary gate. - -Explicit `Is_*` property/value assignments now normalize before built-in -routing, including Perl's colon delimiter. The focused 12-assertion oracle -passes on system Perl, JVM, and interpreter. The generated corpus rises by -exactly 44,944 assertions to 220,712/290,912 on both execution backends, with -identical per-file counts and zero errors, timeouts, or incomplete files. +The current integration stack is preserved through draft PR #1023. It includes +the completed callback/runtime slices, lossless generated Unicode fixtures, +explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, +and the first 524 lines of retired Java-only preprocessor code. Every published +slice has a warning-free combined `make` checkpoint. + +Lexical `use bytes` now compiles non-ASCII substitution patterns with a +single-byte Joni encoding while preserving upgraded, byte-backed, and compiled +`qr//` source provenance. The focused oracle passes 12/12 on system Perl, JVM, +and interpreter, and the exact upstream marker-stage reducer improves from 2/4 +to 4/4 on both execution backends. Generated chunks 05–10 consequently execute +239,843 genuine boundary assertions rather than matching literal UTF-8 marker +text. JVM and interpreter have exact per-file parity at 2,192/239,843 with every +plan complete, exit 0, and no child timeout; all current passes are in the GCB +chunk (2,192/14,953), while the line, sentence, and word boundary chunks expose +zero passing assertions. The runner classifies zero-pass files as `error`, but +their recorded plans, actual counts, and process exits are complete. + +The most recent exact property chunks 01–04 remain 98,092/167,501 on both +execution backends. Combined with the newly authoritative boundary chunks, the +current generated evidence is 100,284/407,344. A current-head refresh of chunks +01–04 is running in a reduced resource lane because an oversubscribed first +attempt reached the runner's no-output deadline; retain the prior exact counts +until that refresh completes. Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while @@ -346,6 +341,10 @@ matcher-specific timeouts on both execution backends. and substitution, including Unicode subjects and code replacements. The focused oracle passes 12/12 and unchanged `subst.t` passes 275/281 on JVM and interpreter. + - [x] Added a provenance-aware single-byte Joni pattern/input path for + non-ASCII substitutions under lexical `use bytes`. Upgraded, byte-backed, + and compiled byte-backed patterns pass 12/12 on all runtimes, and the + generated Unicode marker stage passes 4/4 on JVM and interpreter. - [x] Phase 2: Conditions and backtracking-visible state (2026-08-17) - [x] Implemented executable callback conditions, control verbs including `(*MARK:NAME)`, and callback-visible recursive capture state in Joni. @@ -427,9 +426,10 @@ matcher-specific timeouts on both execution backends. - [x] Rejected 40 invalid Perl inline option/group-name forms in forked Joni with exact JVM/interpreter `reg_mesg.t` parity, reducing residual Joni-only acceptance differences from 198 to 158 (`028602adc`). - - [ ] Fix byte-mode substitution of upgraded marker regexes so chunks 05–10 - exercise real boundary subjects, then close the property and boundary - failures with exact JVM/interpreter plan and semantic parity before marking + - [x] Fixed byte-mode substitution of upgraded marker regexes so chunks 05–10 + exercise real boundary subjects with exact JVM/interpreter plans. + - [ ] Implement native Joni GCB, line, sentence, and word boundary algorithms, + then close the remaining property and boundary failures before marking Phase 3 complete. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) @@ -457,20 +457,23 @@ matcher-specific timeouts on both execution backends. - [x] Removed the disabled invalid-brace pass and its exclusive helpers (`4be6a48e3`; 208 preprocessor lines removed), retaining active Perl/Joni quantifier diagnostics as explicit focused hard/TODO gates. + - [x] Removed the Java-only terminated-whitespace possessification pass + (`c5343aca2`; 80 preprocessor lines removed) after greedy backtracking and + 20,000-character gates passed default and forced-Java policy on both + execution backends. - [ ] Phase 6: Integration and release ### Next Steps -1. Publish the integrated PRUNE and dead invalid-brace preprocessor retirement - slice after its warning-free combined `make`, then integrate the validated - forked-Joni braced-octal diagnostics/tokenization slice. Continue retiring - independently proven Java-only preprocessor rewrites as they complete. -2. Fix lexical `use bytes` substitution when an upgraded marker regex matches a - byte subject, without patching generated fixtures. Regenerate the lossless - corpus and prove that chunks 05–10 no longer match literal boundary markers - before treating any boundary or folding result as evidence. Retain the now- - completed eval byte-source UTF-8 oracle as an independent runtime regression - gate; it is not the generated-corpus marker fix. +1. Publish the byte-mode Joni pattern slice after its warning-free `make` and + authoritative boundary-corpus measurement. Then integrate the validated + braced-octal, braced-hex, and terminated-whitespace slices as focused + stacked PRs; add lazy-negated-class retirement after its validation completes. +2. Implement boundary semantics natively in Joni in measured order: GCB first + (2,192/14,953 currently pass), followed by line, sentence, and word breaks + (currently 0/224,890). Use bundled Perl 5.44 Unicode data and the generated + chunks as the release oracle; remove the simplified Java boundary rewrites + only after native parity. 3. Implement the remaining property clusters in measured order: Block, Script/Script_Extensions, Numeric_Value, Joining_Group, General_Category, break-property values, and Age/In/Present_In. Preserve pinned Perl 5.44 diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index f5a0e7c8eb..bd69e8e741 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -371,7 +371,7 @@ my @copy = @{$z}; # ERROR - ✅ **Inline comments**: `(?#comment)` in regex is implemented. - ✅ **caret modifier**: `(?^` embedded pattern-match modifier, shorthand equivalent to "d-imnsx". - ✅ **\b inside character class**: `[\b]` is supported in regex. -- 🟡 **Unicode boundary assertions**: `\b{gcb}`, `\B{gcb}`, and the `wb`/`lb`/`sb` forms are recognized, but the genuine hand-written generated-corpus preamble still has 25 semantic failures; comprehensive generated coverage is blocked by byte-mode marker normalization. +- 🟡 **Unicode boundary assertions**: `\b{gcb}`, `\B{gcb}`, and the `wb`/`lb`/`sb` forms are recognized. Byte-mode marker normalization is fixed, so the generated boundary corpus is authoritative: GCB passes 2,192/14,953, while line, sentence, and word boundary chunks currently pass 0/224,890 on both execution backends. - ✅ **Variable Interpolation in Regex**: Features like `${var}` for embedding variables. - ✅ **Non-capturing groups**: `(?:...)` is implemented. - ✅ **Named Capture Groups**: Defining named capture groups using `(?...)` or `(?'name'...)` is supported. @@ -386,7 +386,7 @@ my @copy = @{$z}; # ERROR - ✅ **Overloading**: `qr` overloading is implemented. See also [overload pragma](#pragmas). - ❌ **Python-style named groups**: `(?P...)` and `(?P=name)` are accepted by Perl but are not yet parsed by Joni. - ❌ **Alpha assertion aliases**: `(*pla:...)`, `(*plb:...)`, `(*nla:...)`, `(*nlb:...)`, and `(*atomic:...)` are not yet parsed by Joni. -- ❌ **Underscored numeric regex escapes**: Perl spellings such as `\x{0_0_4_1}` and `\o{0_0_1_0_1}` are not yet parsed by Joni; braced octal syntax also needs complete validation and diagnostics. +- ❌ **Underscored numeric regex escapes**: Perl spellings such as `\x{0_0_4_1}` and `\o{0_0_1_0_1}` are not yet parsed by Joni. Valid braced octal and unconditional missing-close/empty diagnostics are implemented in the pending parser stack; three `use re 'strict'` non-octal cases remain deferred. - ✅ **Dynamically-scoped regex variables**: Provisional captures, `$^R`, `$^N`, match positions, and callback locals follow matcher paths and unwind on backtracking. - ✅ **Recursive and Dynamic Patterns**: `(?R)`, `(?0)`, and runtime `(??{ code })` execute through Joni. Dynamic expressions may return strings or `qr//` values, nested alternatives participate in outer backtracking without changing outer grouping or capture numbering, and callback and pure-pattern recursion have engine-owned depth ceilings. @@ -397,8 +397,8 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. The lossless generated `uniprops*.t` matrix currently passes 220,712/290,912 identically on JVM and interpreter, but chunks 01–04 still expose broad loose property/value alias gaps and chunks 05–10 are not yet valid boundary evidence because byte-mode substitution does not normalize upgraded boundary-marker regexes against byte subjects. -- 🟡 **Extended Grapheme Clusters**: Focused `\X` combining-sequence and emoji-ZWJ cases pass, but comprehensive generated UAX verification remains blocked by byte-mode boundary-marker normalization. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 100,284/407,344 identically on JVM and interpreter: chunks 01–04 expose property/value alias gaps, and authoritative chunks 05–10 expose native boundary-algorithm gaps. +- 🟡 **Extended Grapheme Clusters**: Focused `\X` combining-sequence and emoji-ZWJ cases pass. The authoritative generated GCB matrix now passes 2,192/14,953 identically on JVM and interpreter; full native GCB boundary rules remain incomplete. - ✅ **Embedded Code in Regex**: `(?{ code })`, optimistic callbacks `(*{ code })`, executable callback conditions, and `(??{ code })` run as lexical closures in Joni with provisional captures and backtracking unwind. Callback `local` frames follow matcher paths, and escaped loop control or `goto` stops at the callback pseudo-block boundary. `$^N` follows capture-close order independently of `$+`. - ✅ **Regex Debugging**: Lexically scoped `use/no re 'debug'` and `debugcolor` are supported, including runtime snapshot ownership. - ✅ **Runtime Regex Evaluation**: `use re 'eval'` controls whether interpolated patterns containing eval groups may compile. Admitted runtime source is compiled into lexical callback closures and preserves its package, visible lexical cells, Unicode or byte source type, default regex modifiers, and match-once state. Literal callbacks, interpolated `qr//` values (including local, referenced, and tied arrays), raw runtime eval groups, callback conditions, and standalone `(?(DEFINE)...)` containers may be composed in one Joni pattern; dynamic callbacks may return further admitted executable source. diff --git a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java index 5028248612..434938f6c6 100644 --- a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java +++ b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java @@ -1,5 +1,6 @@ package org.perlonjava.runtime.regex; +import org.jcodings.specific.ISO8859_1Encoding; import org.jcodings.specific.UTF8Encoding; import org.joni.Matcher; import org.joni.CalloutHandler; @@ -32,6 +33,7 @@ */ final class JoniRegexPattern { private static final Map INPUT_ENCODINGS = new WeakHashMap<>(); + private static final Map BYTE_INPUT_ENCODINGS = new WeakHashMap<>(); // Ruby syntax defaults \w to ASCII even for a Unicode encoding. Perl's // default and /u modes use Unicode character classes; /a adds ASCII_RANGE @@ -52,6 +54,7 @@ final class JoniRegexPattern { private final RegexFlags flags; private final boolean hasControlVerbState; private final boolean hasDeferredUserDefinedUnicodeProperty; + private final boolean byteMode; JoniRegexPattern(String perlPattern, RegexFlags flags) { this(perlPattern, flags, 0, false); @@ -63,14 +66,23 @@ final class JoniRegexPattern { JoniRegexPattern(String perlPattern, RegexFlags flags, int trustedCalloutCount, boolean forceAsciiClasses) { + this(perlPattern, flags, trustedCalloutCount, forceAsciiClasses, false, false); + } + + JoniRegexPattern(String perlPattern, RegexFlags flags, int trustedCalloutCount, + boolean forceAsciiClasses, boolean byteMode, + boolean byteBackedPattern) { this.flags = flags; + this.byteMode = byteMode; hasControlVerbState = hasControlVerbState(perlPattern); UserPropertyTranslation userProperties = translateUserDefinedProperties(perlPattern, flags); hasDeferredUserDefinedUnicodeProperty = userProperties.deferred(); sourcePattern = translatePattern(userProperties.pattern(), flags, trustedCalloutCount); - byte[] bytes = sourcePattern.getBytes(StandardCharsets.UTF_8); + byte[] bytes = sourcePattern.getBytes(byteMode && byteBackedPattern + ? StandardCharsets.ISO_8859_1 : StandardCharsets.UTF_8); regex = new Regex(bytes, 0, bytes.length, toJoniOptions(flags, forceAsciiClasses), - UTF8Encoding.INSTANCE, PERLONJAVA_SYNTAX); + byteMode ? ISO8859_1Encoding.INSTANCE : UTF8Encoding.INSTANCE, + PERLONJAVA_SYNTAX); namedGroups = collectNamedGroups(regex); } @@ -81,7 +93,7 @@ RegexMatcher matcher(String input, List callbacks) { RegexMatcher matcher(String input, List callbacks, RuntimeScalar subject) { return new JoniRegexMatcher(regex, sourcePattern, namedGroups, flags, - hasControlVerbState, input, callbacks, subject); + hasControlVerbState, byteMode, input, callbacks, subject); } record InputEncoding(byte[] bytes, int[] charToByte, int[] byteToChar) {} @@ -92,6 +104,20 @@ static InputEncoding inputEncoding(String input) { } } + static InputEncoding byteInputEncoding(String input) { + synchronized (BYTE_INPUT_ENCODINGS) { + return BYTE_INPUT_ENCODINGS.computeIfAbsent(input, + JoniRegexPattern::buildByteInputEncoding); + } + } + + private static InputEncoding buildByteInputEncoding(String input) { + byte[] bytes = input.getBytes(StandardCharsets.ISO_8859_1); + int[] identity = new int[input.length() + 1]; + for (int i = 0; i < identity.length; i++) identity[i] = i; + return new InputEncoding(bytes, identity, identity); + } + private static InputEncoding buildInputEncoding(String input) { byte[] bytes = input.getBytes(StandardCharsets.UTF_8); int[] charToByte = JoniRegexMatcher.buildCharToByte(input); @@ -634,22 +660,26 @@ private static final class JoniRegexMatcher implements RegexMatcher { private boolean matched; private int committedLastClosedCapture = -1; private final boolean hasControlVerbState; + private final boolean byteMode; private final List callbacks; private final RuntimeScalar subject; private PerlCalloutHandler calloutHandler; JoniRegexMatcher(Regex regex, String sourcePattern, Map namedGroups, - RegexFlags flags, boolean hasControlVerbState, String input, + RegexFlags flags, boolean hasControlVerbState, boolean byteMode, + String input, List callbacks, RuntimeScalar subject) { this.regex = regex; this.sourcePattern = sourcePattern; this.namedGroups = namedGroups; this.flags = flags; this.hasControlVerbState = hasControlVerbState; + this.byteMode = byteMode; this.input = input; this.callbacks = callbacks; this.subject = subject; - InputEncoding encoding = inputEncoding(input); + InputEncoding encoding = byteMode + ? byteInputEncoding(input) : inputEncoding(input); this.bytes = encoding.bytes(); this.charToByte = encoding.charToByte(); this.byteToChar = encoding.byteToChar(); @@ -797,12 +827,12 @@ private int namedGroupNumber(String name) { if (knownGroup != null) return knownGroup; byte[] nameBytes = name.getBytes(StandardCharsets.UTF_8); return regex.nameToBackrefNumber(nameBytes, 0, nameBytes.length, - UTF8Encoding.INSTANCE, captures); + byteMode ? ISO8859_1Encoding.INSTANCE : UTF8Encoding.INSTANCE, captures); } private int advanceCodePoint(int offset) { return offset >= regionEnd ? regionEnd + 1 - : offset + Character.charCount(input.codePointAt(offset)); + : offset + (byteMode ? 1 : Character.charCount(input.codePointAt(offset))); } private int toCharOffset(int byteOffset) { diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java index 60772dbc15..66491c48eb 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java @@ -186,12 +186,16 @@ static void updateControlVerbVariables(String mark, String error) { Pattern notemptyPatternUnicode; JoniRegexPattern recursivePattern; JoniRegexPattern recursivePatternUnicode; + JoniRegexPattern recursivePatternBytes; List executableCallbacks = List.of(); private boolean executableCallbacksReleased; int[] branchResetCaptureMap; int patternFlags; int patternFlagsUnicode; public String patternString; + // Source scalar provenance needed when a compiled qr// is reused under + // lexical use bytes: byte-backed source characters are already octets. + private boolean patternByteBacked; String javaPatternString; // Preprocessed Java-compatible pattern for recompilation private String requiredLiteral; boolean hasPreservesMatch = false; // True if /p was used (outer or inline (?p)) @@ -241,11 +245,13 @@ public RuntimeRegex cloneTracked() { copy.notemptyPatternUnicode = this.notemptyPatternUnicode; copy.recursivePattern = this.recursivePattern; copy.recursivePatternUnicode = this.recursivePatternUnicode; + copy.recursivePatternBytes = this.recursivePatternBytes; copy.setExecutableCallbacks(this.executableCallbacks); copy.branchResetCaptureMap = this.branchResetCaptureMap; copy.patternFlags = this.patternFlags; copy.patternFlagsUnicode = this.patternFlagsUnicode; copy.patternString = this.patternString; + copy.patternByteBacked = this.patternByteBacked; copy.javaPatternString = this.javaPatternString; copy.requiredLiteral = this.requiredLiteral; copy.hasPreservesMatch = this.hasPreservesMatch; @@ -313,6 +319,10 @@ public RegexMatcher matcher(RuntimeScalar string, String input) { } private JoniRegexPattern selectRecursivePattern(RuntimeScalar string) { + if (bytesSubstitution && recursivePatternBytes != null + && string.type == RuntimeScalarType.BYTE_STRING) { + return recursivePatternBytes; + } if (recursivePatternUnicode != null && recursivePatternUnicode != recursivePattern && regexFlags != null && !regexFlags.isAscii() && Utf8.isUtf8(string)) { return recursivePatternUnicode; @@ -1373,6 +1383,7 @@ && containsExecutableSource(patternString.toString(), modifierStr.indexOf('x') > regex.patternNoInternalMarkers = originalRegex.patternNoInternalMarkers; regex.patternUnicodeNoInternalMarkers = originalRegex.patternUnicodeNoInternalMarkers; regex.patternString = originalRegex.patternString; + regex.patternByteBacked = originalRegex.patternByteBacked; regex.deferredUserDefinedUnicodeProperties = originalRegex.deferredUserDefinedUnicodeProperties; regex.hasPreservesMatch = originalRegex.hasPreservesMatch; @@ -1415,6 +1426,7 @@ && containsExecutableSource(patternString.toString(), modifierStr.indexOf('x') > regex.patternNoInternalMarkers = originalRegex.patternNoInternalMarkers; regex.patternUnicodeNoInternalMarkers = originalRegex.patternUnicodeNoInternalMarkers; regex.patternString = originalRegex.patternString; + regex.patternByteBacked = originalRegex.patternByteBacked; regex.deferredUserDefinedUnicodeProperties = originalRegex.deferredUserDefinedUnicodeProperties; regex.hasPreservesMatch = originalRegex.hasPreservesMatch; @@ -1445,8 +1457,10 @@ && containsExecutableSource(patternString.toString(), modifierStr.indexOf('x') > // Default: compile as string (cloneTracked() creates a tracked copy // so the cached RuntimeRegex is not corrupted by refCount changes) - return new RuntimeScalar(compile(patternString.toString(), modifierStr, callSiteDebugMode).cloneTracked()) - .propagateTaint(patternString); + RuntimeRegex compiled = compile(patternString.toString(), modifierStr, + callSiteDebugMode).cloneTracked(); + compiled.patternByteBacked = patternString.type == RuntimeScalarType.BYTE_STRING; + return new RuntimeScalar(compiled).propagateTaint(patternString); } static boolean containsExecutableSource(String pattern) { @@ -1617,6 +1631,7 @@ public static RuntimeScalar getReplacementRegex(RuntimeScalar patternString, Run regex.patternNoInternalMarkers = resolvedRegex.patternNoInternalMarkers; regex.patternUnicodeNoInternalMarkers = resolvedRegex.patternUnicodeNoInternalMarkers; regex.patternString = resolvedRegex.patternString; + regex.patternByteBacked = resolvedRegex.patternByteBacked; regex.deferredUserDefinedUnicodeProperties = resolvedRegex.deferredUserDefinedUnicodeProperties; regex.regexFlags = resolvedRegex.regexFlags; @@ -1705,10 +1720,28 @@ public static RuntimeScalar getBytesReplacementRegex(RuntimeScalar patternString RuntimeScalar modifiers, RuntimeArray callerArgs) { RuntimeScalar result = getReplacementRegex(patternString, replacement, modifiers, callerArgs); - ((RuntimeRegex) result.value).bytesSubstitution = true; + RuntimeRegex regex = (RuntimeRegex) result.value; + regex.bytesSubstitution = true; + String sourcePattern = patternString.type == RuntimeScalarType.REGEX + ? regex.patternString : patternString.toString(); + if (regex.recursivePattern != null && containsNonAscii(sourcePattern)) { + boolean byteBackedPattern = patternString.type == RuntimeScalarType.REGEX + ? regex.patternByteBacked + : patternString.type == RuntimeScalarType.BYTE_STRING; + regex.recursivePatternBytes = new JoniRegexPattern(sourcePattern, + regex.regexFlags, regex.executableCallbacks.size(), true, + true, byteBackedPattern); + } return result; } + private static boolean containsNonAscii(String value) { + for (int i = 0; i < value.length(); i++) { + if (value.charAt(i) > 0x7f) return true; + } + return false; + } + /** * Applies a Perl "qr" object on a string; returns true/false or a list, * and produces side-effects. diff --git a/src/test/resources/unit/regex/bytes_upgraded_pattern_substitution.t b/src/test/resources/unit/regex/bytes_upgraded_pattern_substitution.t new file mode 100644 index 0000000000..7b70677409 --- /dev/null +++ b/src/test/resources/unit/regex/bytes_upgraded_pattern_substitution.t @@ -0,0 +1,43 @@ +use strict; +use warnings; +use Test::More tests => 12; + +my $marker_utf8 = my $marker = chr(0xF7); +utf8::upgrade($marker_utf8); + +my $bytes = pack('C*', 0x27, 0xC3, 0xB7, 0x27); +ok(!utf8::is_utf8($bytes), 'fixture starts as an octet string'); +is(length($bytes), 4, 'fixture contains the encoded marker bytes'); + +{ + use bytes; + is($bytes =~ s/$marker_utf8/$marker/g, 1, + 'upgraded pattern matches its encoded octets under use bytes'); +} + +ok(!utf8::is_utf8($bytes), 'byte substitution preserves octet storage'); +is(length($bytes), 3, 'encoded marker contracts to one replacement octet'); +is(unpack('H*', $bytes), '27f727', 'replacement has the canonical marker byte'); + +my $byte_pattern = pack('C*', 0xC3, 0xB7); +my $byte_pattern_target = pack('C*', 0x41, 0xC3, 0xB7, 0x42); +{ + use bytes; + is($byte_pattern_target =~ s/$byte_pattern/X/g, 1, + 'byte-backed pattern remains a literal octet sequence'); +} +is($byte_pattern_target, 'AXB', 'byte-backed pattern consumes exactly two octets'); + +my $byte_qr = qr/$byte_pattern/; +my $byte_qr_target = pack('C*', 0x41, 0xC3, 0xB7, 0x42); +{ + use bytes; + is($byte_qr_target =~ s/$byte_qr/Y/g, 1, + 'compiled byte-backed pattern retains its octet provenance'); +} +is($byte_qr_target, 'AYB', 'compiled byte-backed pattern consumes two octets'); + +my $ordinary = "A${marker_utf8}B"; +is($ordinary =~ s/$marker_utf8/X/g, 1, + 'ordinary Unicode substitution still matches one character'); +is($ordinary, 'AXB', 'ordinary Unicode substitution remains character based'); From c087a94315fc55c04252bfa01b028ce747a52e4d Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:17 +0200 Subject: [PATCH 20/60] feat(regex): implement native Joni grapheme boundaries (#1025) Squashed integration of PR #1025 at 12360ac99. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 72 +++++++----- docs/reference/feature-matrix.md | 6 +- .../unit/regex/grapheme_boundary_rules.t | 62 +++++++++++ .../joni/src/org/joni/ArrayCompiler.java | 8 ++ .../joni/src/org/joni/ByteCodeMachine.java | 105 ++++++++++++++++++ third_party/joni/src/org/joni/Lexer.java | 30 +++++ third_party/joni/src/org/joni/Parser.java | 20 +++- .../joni/constants/internal/AnchorType.java | 6 + .../org/joni/constants/internal/OPCode.java | 6 + .../joni/test/TestPerlGraphemeBoundary.java | 61 ++++++++++ 10 files changed, 346 insertions(+), 30 deletions(-) create mode 100644 src/test/resources/unit/regex/grapheme_boundary_rules.t create mode 100644 third_party/joni/test/org/joni/test/TestPerlGraphemeBoundary.java diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 2316220f3c..e80850ad78 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,7 +224,7 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The current integration stack is preserved through draft PR #1023. It includes +The current integration stack is preserved through review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, and the first 524 lines of retired Java-only preprocessor code. Every published @@ -237,17 +237,22 @@ and interpreter, and the exact upstream marker-stage reducer improves from 2/4 to 4/4 on both execution backends. Generated chunks 05–10 consequently execute 239,843 genuine boundary assertions rather than matching literal UTF-8 marker text. JVM and interpreter have exact per-file parity at 2,192/239,843 with every -plan complete, exit 0, and no child timeout; all current passes are in the GCB -chunk (2,192/14,953), while the line, sentence, and word boundary chunks expose -zero passing assertions. The runner classifies zero-pass files as `error`, but -their recorded plans, actual counts, and process exits are complete. +plan complete, exit 0, and no child timeout at the pre-GCB baseline. The runner +classifies zero-pass files as `error`, but their recorded plans, actual counts, +and process exits are complete. + +Native Joni GCB assertions now implement GB1–GB13 and GB999, including Indic +conjunct and emoji-ZWJ context, and `\X` consumes repeated GB9c conjuncts. The +focused 29-assertion oracle passes on system Perl, JVM, and interpreter. +Authoritative chunk 05 improves by 6,324 assertions from 2,192/14,953 to +8,516/14,976 identically on JVM and interpreter: its complete GCB/`\X` section +passes, leaving only the 6,460 sentence-boundary assertions in that chunk. The most recent exact property chunks 01–04 remain 98,092/167,501 on both -execution backends. Combined with the newly authoritative boundary chunks, the -current generated evidence is 100,284/407,344. A current-head refresh of chunks -01–04 is running in a reduced resource lane because an oversubscribed first -attempt reached the runner's no-output deadline; retain the prior exact counts -until that refresh completes. +execution backends. Combined with the native-GCB result and unchanged chunks +06–10, current generated evidence is 106,608/407,367. A resource-contended +current-head refresh did not reproduce a complete exact JVM/interpreter pair, +so it does not replace that accepted baseline. Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while @@ -345,6 +350,10 @@ matcher-specific timeouts on both execution backends. non-ASCII substitutions under lexical `use bytes`. Upgraded, byte-backed, and compiled byte-backed patterns pass 12/12 on all runtimes, and the generated Unicode marker stage passes 4/4 on JVM and interpreter. + - [ ] Close `/g` same-position retry and capture semantics after a zero-width + first alternative. The standard-Perl omniholder reducer currently passes + 7/10 in all PerlOnJava modes even though the DBIx substitution contract + passes; this is independent of the Java-only alternative-reorder rewrite. - [x] Phase 2: Conditions and backtracking-visible state (2026-08-17) - [x] Implemented executable callback conditions, control verbs including `(*MARK:NAME)`, and callback-visible recursive capture state in Joni. @@ -426,11 +435,18 @@ matcher-specific timeouts on both execution backends. - [x] Rejected 40 invalid Perl inline option/group-name forms in forked Joni with exact JVM/interpreter `reg_mesg.t` parity, reducing residual Joni-only acceptance differences from 198 to 158 (`028602adc`). + - [x] Validated native Python-style named captures and backreferences plus + removal of their frontend conversion (`afbe2bc34`). The 20-case oracle + passes on both execution backends with exact malformed/unknown diagnostics; + integration into the coordinator stack remains queued. - [x] Fixed byte-mode substitution of upgraded marker regexes so chunks 05–10 exercise real boundary subjects with exact JVM/interpreter plans. - - [ ] Implement native Joni GCB, line, sentence, and word boundary algorithms, - then close the remaining property and boundary failures before marking - Phase 3 complete. + - [x] Implemented native Joni GCB assertions for GB1–GB13 and GB999 and aligned + `\X` with repeated GB9c Indic conjunct behavior. The focused oracle passes + 29/29 and generated chunk 05 reaches 8,516/14,976 on both execution backends. + - [ ] Implement native Joni line, sentence, and word boundary algorithms, then + close the remaining property and boundary failures before marking Phase 3 + complete. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) - [x] Preserved mixed executable-source provenance, nested dynamic callback @@ -461,26 +477,32 @@ matcher-specific timeouts on both execution backends. (`c5343aca2`; 80 preprocessor lines removed) after greedy backtracking and 20,000-character gates passed default and forced-Java policy on both execution backends. + - [x] Validated removal of the Java-only terminated lazy-negated-class + possessification pass (`625ea97a2`; 252 preprocessor lines removed) with + leftmost-capture and 20,000-character gates in all four backend modes. + - [x] Retired the Java-only DBIx omniholder alternative reorder + (`18e71a532`; 50 lines removed) after exact substitution and bundled + DBIx::Simple gates passed without changing the separately tracked raw `/g` + 7/10 result. - [ ] Phase 6: Integration and release ### Next Steps -1. Publish the byte-mode Joni pattern slice after its warning-free `make` and - authoritative boundary-corpus measurement. Then integrate the validated - braced-octal, braced-hex, and terminated-whitespace slices as focused - stacked PRs; add lazy-negated-class retirement after its validation completes. -2. Implement boundary semantics natively in Joni in measured order: GCB first - (2,192/14,953 currently pass), followed by line, sentence, and word breaks - (currently 0/224,890). Use bundled Perl 5.44 Unicode data and the generated - chunks as the release oracle; remove the simplified Java boundary rewrites - only after native parity. +1. Land review-ready PR #1024, publish the native-GCB slice as its own stacked + PR, then integrate the validated braced-octal, braced-hex, + terminated-whitespace, and lazy-negated-class slices as focused stacked PRs. +2. Continue boundary semantics natively in Joni in measured order: sentence + breaks next (0/6,460 in the remainder of chunk 05), followed by line and word + breaks (0/224,890 in chunks 06–10). Use bundled Perl 5.44 Unicode data and + the generated chunks as the release oracle; remove each simplified Java + boundary rewrite only after native parity. 3. Implement the remaining property clusters in measured order: Block, Script/Script_Extensions, Numeric_Value, Joining_Group, General_Category, break-property values, and Age/In/Present_In. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. - Close the independently reduced Joni syntax gaps for Python-style named - groups, alpha assertion aliases, underscored numeric escapes, and braced - octal parsing with focused standard-Perl gates. + Integrate the validated Python-style named-group and braced-octal slices, + then close alpha assertion aliases and underscored numeric escapes with + focused standard-Perl gates. 4. Rerun the forced-Joni 80-file corpus on JVM and interpreter from the combined head. Save complete JSON and logs, publish the missing differential report, and compare every file with both the Phase 0 result and PR 958 under the diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index bd69e8e741..1eeea4410e 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -371,7 +371,7 @@ my @copy = @{$z}; # ERROR - ✅ **Inline comments**: `(?#comment)` in regex is implemented. - ✅ **caret modifier**: `(?^` embedded pattern-match modifier, shorthand equivalent to "d-imnsx". - ✅ **\b inside character class**: `[\b]` is supported in regex. -- 🟡 **Unicode boundary assertions**: `\b{gcb}`, `\B{gcb}`, and the `wb`/`lb`/`sb` forms are recognized. Byte-mode marker normalization is fixed, so the generated boundary corpus is authoritative: GCB passes 2,192/14,953, while line, sentence, and word boundary chunks currently pass 0/224,890 on both execution backends. +- 🟡 **Unicode boundary assertions**: `\b{gcb}` and `\B{gcb}` implement GB1–GB13 and GB999 natively in Joni; all 6,285 generated GCB boundary assertions pass on both execution backends. The recognized `sb`/`lb`/`wb` forms still use simplified frontend rewrites and pass 0/231,350 authoritative generated assertions. - ✅ **Variable Interpolation in Regex**: Features like `${var}` for embedding variables. - ✅ **Non-capturing groups**: `(?:...)` is implemented. - ✅ **Named Capture Groups**: Defining named capture groups using `(?...)` or `(?'name'...)` is supported. @@ -397,8 +397,8 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 100,284/407,344 identically on JVM and interpreter: chunks 01–04 expose property/value alias gaps, and authoritative chunks 05–10 expose native boundary-algorithm gaps. -- 🟡 **Extended Grapheme Clusters**: Focused `\X` combining-sequence and emoji-ZWJ cases pass. The authoritative generated GCB matrix now passes 2,192/14,953 identically on JVM and interpreter; full native GCB boundary rules remain incomplete. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 106,608/407,367 identically on JVM and interpreter: chunks 01–04 expose property/value alias gaps, and the remaining boundary failures are sentence, line, and word breaks. +- ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. - ✅ **Embedded Code in Regex**: `(?{ code })`, optimistic callbacks `(*{ code })`, executable callback conditions, and `(??{ code })` run as lexical closures in Joni with provisional captures and backtracking unwind. Callback `local` frames follow matcher paths, and escaped loop control or `goto` stops at the callback pseudo-block boundary. `$^N` follows capture-close order independently of `$+`. - ✅ **Regex Debugging**: Lexically scoped `use/no re 'debug'` and `debugcolor` are supported, including runtime snapshot ownership. - ✅ **Runtime Regex Evaluation**: `use re 'eval'` controls whether interpolated patterns containing eval groups may compile. Admitted runtime source is compiled into lexical callback closures and preserves its package, visible lexical cells, Unicode or byte source type, default regex modifiers, and match-once state. Literal callbacks, interpolated `qr//` values (including local, referenced, and tied arrays), raw runtime eval groups, callback conditions, and standalone `(?(DEFINE)...)` containers may be composed in one Joni pattern; dynamic callbacks may return further admitted executable source. diff --git a/src/test/resources/unit/regex/grapheme_boundary_rules.t b/src/test/resources/unit/regex/grapheme_boundary_rules.t new file mode 100644 index 0000000000..c5f0b84aa8 --- /dev/null +++ b/src/test/resources/unit/regex/grapheme_boundary_rules.t @@ -0,0 +1,62 @@ +use strict; +use warnings; +use utf8; +use Test::More tests => 29; + +sub boundary_at { + my ($subject, $offset) = @_; + my $left = quotemeta substr($subject, 0, $offset); + my $right = quotemeta substr($subject, $offset); + return $subject =~ /\A${left}\b{gcb}${right}\z/; +} + +sub no_boundary_at { + my ($subject, $offset) = @_; + my $left = quotemeta substr($subject, 0, $offset); + my $right = quotemeta substr($subject, $offset); + return $subject =~ /\A${left}\B{gcb}${right}\z/; +} + +ok(boundary_at('ab', 0), 'GB1 start of text is a boundary'); +ok(boundary_at('ab', 2), 'GB2 end of text is a boundary'); +ok(no_boundary_at("\r\n", 1), 'GB3 keeps CR LF together'); +ok(boundary_at("\rA", 1), 'GB4 breaks after CR'); +ok(boundary_at("A\n", 1), 'GB5 breaks before LF'); +ok(no_boundary_at("\x{1100}\x{1161}", 1), 'GB6 keeps Hangul L before V'); +ok(no_boundary_at("\x{AC00}\x{11A8}", 1), 'GB7 keeps Hangul LV before T'); +ok(no_boundary_at("\x{AC01}\x{11A8}", 1), 'GB8 keeps Hangul LVT before T'); +ok(no_boundary_at("A\x{0308}", 1), 'GB9 keeps Extend with its base'); +ok(no_boundary_at("A\x{200D}", 1), 'GB9 keeps ZWJ with its base'); +ok(no_boundary_at("A\x{0903}", 1), 'GB9a keeps SpacingMark with its base'); +ok(no_boundary_at("\x{0600}A", 1), 'GB9b keeps Prepend with following text'); +ok(no_boundary_at("\x{0915}\x{094D}\x{0915}", 2), + 'GB9c keeps Indic conjunct linker sequence together'); +ok(no_boundary_at("\x{1F469}\x{0308}\x{200D}\x{1F680}", 3), + 'GB11 keeps extended pictograph ZWJ sequence together'); +ok(no_boundary_at("\x{1F1E6}\x{1F1E7}", 1), + 'GB12 keeps the first regional-indicator pair together'); +ok(boundary_at("\x{1F1E6}\x{1F1E7}\x{1F1E8}", 2), + 'GB12 breaks before an odd trailing regional indicator'); +ok(boundary_at('AB', 1), 'GB999 breaks between ordinary letters'); +ok(!no_boundary_at('AB', 1), 'negated assertion rejects a real boundary'); + +for my $subject ( + "\x{0915}\x{094D}\x{0924}", + "\x{0915}\x{094D}\x{094D}\x{0924}", + "\x{0915}\x{094D}\x{200D}\x{0924}", + "\x{0915}\x{093C}\x{200D}\x{094D}\x{0924}", + "\x{0915}\x{094D}\x{0924}\x{094D}\x{092F}", + "\x{0915}\x{094D}\x{200D}\x{0924}\x{094D}\x{200D}\x{092F}", + "\x{1004}\x{103A}\x{1039}\x{1011}\x{1039}\x{1011}", + "\x{1B32}\x{1B44}\x{1B22}\x{1B44}\x{1B2C}", + "\x{179F}\x{17D2}\x{178F}\x{17D2}\x{179A}\x{17B8}", +) { + is(($subject =~ /\A(\X)\z/)[0], $subject, + '\\X consumes the complete GB9c Indic conjunct sequence'); +} + +{ + use bytes; + ok("AB" =~ /\AA\b{gcb}B\z/, 'byte-mode ASCII letters have a GCB boundary'); + ok("\r\n" =~ /\A\r\B{gcb}\n\z/, 'byte-mode CR LF remains one cluster'); +} diff --git a/third_party/joni/src/org/joni/ArrayCompiler.java b/third_party/joni/src/org/joni/ArrayCompiler.java index d4d665e692..7a1ffe5e7a 100644 --- a/third_party/joni/src/org/joni/ArrayCompiler.java +++ b/third_party/joni/src/org/joni/ArrayCompiler.java @@ -1198,6 +1198,14 @@ protected void compileAnchorNode(AnchorNode node) { } break; + case AnchorType.GRAPHEME_BOUNDARY: + addOpcode(OPCode.GRAPHEME_BOUNDARY); + break; + + case AnchorType.NOT_GRAPHEME_BOUNDARY: + addOpcode(OPCode.NOT_GRAPHEME_BOUNDARY); + break; + case AnchorType.WORD_BEGIN: if (Config.USE_WORD_BEGIN_END) { if (node.asciiRange) { diff --git a/third_party/joni/src/org/joni/ByteCodeMachine.java b/third_party/joni/src/org/joni/ByteCodeMachine.java index e69872013c..88739ce6a2 100644 --- a/third_party/joni/src/org/joni/ByteCodeMachine.java +++ b/third_party/joni/src/org/joni/ByteCodeMachine.java @@ -32,6 +32,7 @@ import org.jcodings.CodeRange; import org.jcodings.Encoding; import org.jcodings.IntHolder; +import org.jcodings.unicode.UnicodeCodeRange; import org.joni.constants.internal.OPCode; import org.joni.constants.internal.OPSize; import org.joni.exception.ErrorMessages; @@ -249,6 +250,8 @@ private final int execute(final boolean checkThreadInterrupt) throws Interrupted case OPCode.NOT_WORD: opNotWord(); break; case OPCode.WORD_BOUND: opWordBound(); continue; case OPCode.NOT_WORD_BOUND: opNotWordBound(); continue; + case OPCode.GRAPHEME_BOUNDARY: opGraphemeBoundary(false); continue; + case OPCode.NOT_GRAPHEME_BOUNDARY: opGraphemeBoundary(true); continue; case OPCode.WORD_BEGIN: opWordBegin(); continue; case OPCode.WORD_END: opWordEnd(); continue; @@ -406,6 +409,8 @@ private final int executeSb(final boolean checkThreadInterrupt) throws Interrupt case OPCode.NOT_WORD: opNotWordSb(); break; case OPCode.WORD_BOUND: opWordBoundSb(); continue; case OPCode.NOT_WORD_BOUND: opNotWordBoundSb(); continue; + case OPCode.GRAPHEME_BOUNDARY: opGraphemeBoundary(false); continue; + case OPCode.NOT_GRAPHEME_BOUNDARY: opGraphemeBoundary(true); continue; case OPCode.WORD_BEGIN: opWordBeginSb(); continue; case OPCode.WORD_END: opWordEndSb(); continue; @@ -1238,6 +1243,106 @@ private void opWordBound() { } } + private void opGraphemeBoundary(boolean negated) { + if (isGraphemeBoundary() == negated) opFail(); + } + + private boolean isGraphemeBoundary() { + if (s <= str || s >= end) return true; // GB1, GB2 + + int leftPosition = enc.prevCharHead(bytes, str, s, end); + int left = enc.mbcToCode(bytes, leftPosition, end); + int right = enc.mbcToCode(bytes, s, end); + + if (isGcb(left, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_CR) + && isGcb(right, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_LF)) return false; // GB3 + if (isGcbControl(left) || isGcbControl(right)) return true; // GB4, GB5 + + if (isGcb(left, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_L) + && (isGcb(right, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_L) + || isGcb(right, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_V) + || isGcb(right, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_LV) + || isGcb(right, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_LVT))) return false; // GB6 + if ((isGcb(left, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_LV) + || isGcb(left, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_V)) + && (isGcb(right, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_V) + || isGcb(right, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_T))) return false; // GB7 + if ((isGcb(left, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_LVT) + || isGcb(left, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_T)) + && isGcb(right, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_T)) return false; // GB8 + + if (isGcb(right, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_EXTEND) + || isGcb(right, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_ZWJ)) return false; // GB9 + if (isGcb(right, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_SPACINGMARK)) return false; // GB9a + if (isGcb(left, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_PREPEND)) return false; // GB9b + if (isIndicConjunctBoundary(leftPosition, right)) return false; // GB9c + if (isExtendedPictographicBoundary(leftPosition, right)) return false; // GB11 + if (isRegionalIndicatorBoundary(leftPosition, left, right)) return false; // GB12, GB13 + + return true; // GB999 + } + + private boolean isGcbControl(int codePoint) { + return isGcb(codePoint, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_CR) + || isGcb(codePoint, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_LF) + || isGcb(codePoint, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_CONTROL); + } + + private boolean isGcb(int codePoint, UnicodeCodeRange range) { + return enc.isCodeCType(codePoint, range.getCType()); + } + + private boolean isIndicConjunctBoundary(int leftPosition, int right) { + if (!isGcb(right, UnicodeCodeRange.INCBCONSONANT)) return false; + boolean sawLinker = false; + int position = leftPosition; + while (position >= str) { + int codePoint = enc.mbcToCode(bytes, position, end); + if (isGcb(codePoint, UnicodeCodeRange.INCBLINKER)) { + sawLinker = true; + } else if (!isGcb(codePoint, UnicodeCodeRange.INCBEXTEND)) { + return sawLinker && isGcb(codePoint, UnicodeCodeRange.INCBCONSONANT); + } + if (position == str) break; + position = enc.prevCharHead(bytes, str, position, end); + } + return false; + } + + private boolean isExtendedPictographicBoundary(int leftPosition, int right) { + if (!isGcb(right, UnicodeCodeRange.EXTENDEDPICTOGRAPHIC) + || !isGcb( + enc.mbcToCode(bytes, leftPosition, end), + UnicodeCodeRange.GRAPHEMECLUSTERBREAK_ZWJ)) return false; + if (leftPosition == str) return false; + int position = enc.prevCharHead(bytes, str, leftPosition, end); + while (position >= str) { + int codePoint = enc.mbcToCode(bytes, position, end); + if (!isGcb(codePoint, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_EXTEND)) { + return isGcb(codePoint, UnicodeCodeRange.EXTENDEDPICTOGRAPHIC); + } + if (position == str) break; + position = enc.prevCharHead(bytes, str, position, end); + } + return false; + } + + private boolean isRegionalIndicatorBoundary(int leftPosition, int left, int right) { + if (!isGcb(left, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_REGIONALINDICATOR) + || !isGcb(right, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_REGIONALINDICATOR)) { + return false; + } + int count = 1; + int position = leftPosition; + while (position > str) { + position = enc.prevCharHead(bytes, str, position, end); + int codePoint = enc.mbcToCode(bytes, position, end); + if (!isGcb(codePoint, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_REGIONALINDICATOR)) break; + count++; + } + return (count & 1) == 1; + } + private void opWordBoundSb() { if (s == str) { if (s >= range || !enc.isWord(bytes[s] & 0xff)) {opFail(); return;} diff --git a/third_party/joni/src/org/joni/Lexer.java b/third_party/joni/src/org/joni/Lexer.java index 9b80b48ef3..518261af31 100644 --- a/third_party/joni/src/org/joni/Lexer.java +++ b/third_party/joni/src/org/joni/Lexer.java @@ -1130,12 +1130,14 @@ protected final void fetchToken() { break; case 'b': if (syntax.opEscBWordBound()) { + if (fetchTokenForPerlGraphemeBoundary(false)) break; fetchTokenFor_anchor(AnchorType.WORD_BOUND); token.setAnchorASCIIRange(isAsciiRange(env.option) && !isWordBoundAllRange(env.option)); } break; case 'B': if (syntax.opEscBWordBound()) { + if (fetchTokenForPerlGraphemeBoundary(true)) break; fetchTokenFor_anchor(AnchorType.NOT_WORD_BOUND); token.setAnchorASCIIRange(isAsciiRange(env.option) && !isWordBoundAllRange(env.option)); } @@ -1343,6 +1345,34 @@ protected final void fetchToken() { } // while } + private boolean fetchTokenForPerlGraphemeBoundary(boolean negated) { + if (!syntax.op2OptionPerl() || !left() || peek() != '{') return false; + + mark(); + fetch(); // '{' + StringBuilder name = new StringBuilder(4); + while (left()) { + fetch(); + if (c == '}') { + if (name.toString().equals("gcb")) { + fetchTokenFor_anchor(negated + ? AnchorType.NOT_GRAPHEME_BOUNDARY + : AnchorType.GRAPHEME_BOUNDARY); + return true; + } + restore(); + return false; + } + if (name.length() >= 8 || c > 0x7f) { + restore(); + return false; + } + name.append((char)c); + } + restore(); + return false; + } + private void greedyCheck() { if (left() && peekIs('?') && syntax.opQMarkNonGreedy()) { diff --git a/third_party/joni/src/org/joni/Parser.java b/third_party/joni/src/org/joni/Parser.java index 150b01caac..40971c1cfb 100644 --- a/third_party/joni/src/org/joni/Parser.java +++ b/third_party/joni/src/org/joni/Parser.java @@ -1180,7 +1180,7 @@ private ListNode createNodeFromArray(Node[]nodes, int nodeArray) { return np; } - private static final int NODE_COMMON_SIZE = 16; + private static final int NODE_COMMON_SIZE = 20; private Node parseExtendedGraphemeCluster() { final Node[] nodes = new Node[NODE_COMMON_SIZE]; final int anyTargetPosition; @@ -1247,8 +1247,24 @@ private Node parseExtendedGraphemeCluster() { quantifierNode(nodes, XPList + 1, 0, QuantifierNode.REPEAT_INFINITE); createNodeFromArray(true, nodes, coreAlts + 4, XPList); } + { + int incbList = coreAlts + 8; + createPropertyNode(nodes, incbList + 0, UnicodeCodeRange.INCBCONSONANT); + int conjunctTail = incbList + 2; + createPropertyNode(nodes, conjunctTail + 0, UnicodeCodeRange.INCBEXTEND); + quantifierNode(nodes, conjunctTail + 0, 0, QuantifierNode.REPEAT_INFINITE); + createPropertyNode(nodes, conjunctTail + 1, UnicodeCodeRange.INCBLINKER); + quantifierNode(nodes, conjunctTail + 1, 1, QuantifierNode.REPEAT_INFINITE); + createPropertyNode(nodes, conjunctTail + 2, UnicodeCodeRange.INCBEXTEND); + addPropertyToCC((CClassNode)nodes[conjunctTail + 2], UnicodeCodeRange.INCBLINKER, false); + quantifierNode(nodes, conjunctTail + 2, 0, QuantifierNode.REPEAT_INFINITE); + createPropertyNode(nodes, conjunctTail + 3, UnicodeCodeRange.INCBCONSONANT); + createNodeFromArray(true, nodes, incbList + 1, conjunctTail); + quantifierNode(nodes, incbList + 1, 1, QuantifierNode.REPEAT_INFINITE); + createNodeFromArray(true, nodes, coreAlts + 5, incbList); + } cc = new CClassNode(); - nodes[coreAlts + 5] = cc; + nodes[coreAlts + 6] = cc; if (enc.minLength() > 1) { addPropertyToCC(cc, UnicodeCodeRange.GRAPHEMECLUSTERBREAK_CONTROL, false); cc.addCodeRange(env, 0x000A, 0x000A); diff --git a/third_party/joni/src/org/joni/constants/internal/AnchorType.java b/third_party/joni/src/org/joni/constants/internal/AnchorType.java index 6851c995fd..6b3c845f09 100644 --- a/third_party/joni/src/org/joni/constants/internal/AnchorType.java +++ b/third_party/joni/src/org/joni/constants/internal/AnchorType.java @@ -43,6 +43,8 @@ public interface AnchorType { int END_BUF_MASK = (END_BUF | SEMI_END_BUF); int KEEP = (1<<16); + int GRAPHEME_BOUNDARY = (1<<17); + int NOT_GRAPHEME_BOUNDARY = (1<<18); int ALLOWED_IN_LB = ( LOOK_BEHIND | LOOK_BEHIND_NOT | @@ -53,6 +55,8 @@ public interface AnchorType { KEEP | WORD_BOUND | NOT_WORD_BOUND | + GRAPHEME_BOUNDARY | + NOT_GRAPHEME_BOUNDARY | WORD_BEGIN | WORD_END ); @@ -66,6 +70,8 @@ public interface AnchorType { KEEP | WORD_BOUND | NOT_WORD_BOUND | + GRAPHEME_BOUNDARY | + NOT_GRAPHEME_BOUNDARY | WORD_BEGIN | WORD_END ); } diff --git a/third_party/joni/src/org/joni/constants/internal/OPCode.java b/third_party/joni/src/org/joni/constants/internal/OPCode.java index 6130cabf9b..05a52caa63 100644 --- a/third_party/joni/src/org/joni/constants/internal/OPCode.java +++ b/third_party/joni/src/org/joni/constants/internal/OPCode.java @@ -154,6 +154,8 @@ public interface OPCode { int RECURSION_CONDITION = 110; /* active subpattern call conditional */ int CHECK_POS_END = 111; /* require current position at saved positive boundary */ int CHECK_LOOK_BEHIND_END = 112; /* require current position at saved negative boundary */ + int GRAPHEME_BOUNDARY = 113; /* Perl \b{gcb} */ + int NOT_GRAPHEME_BOUNDARY = 114; /* Perl \B{gcb} */ String[] OpCodeNames = Config.DEBUG_COMPILE ? new String[] { "finish", /*OP_FINISH*/ @@ -270,6 +272,8 @@ public interface OPCode { "recursion-condition", /*OP_RECURSION_CONDITION*/ "check-pos-end", /*OP_CHECK_POS_END*/ "check-look-behind-end", /*OP_CHECK_LOOK_BEHIND_END*/ + "grapheme-boundary", /*OP_GRAPHEME_BOUNDARY*/ + "not-grapheme-boundary", /*OP_NOT_GRAPHEME_BOUNDARY*/ } : null; int[] OpCodeArgTypes = Config.DEBUG_COMPILE ? new int[] { @@ -387,5 +391,7 @@ public interface OPCode { Arguments.SPECIAL, /*OP_RECURSION_CONDITION*/ Arguments.NON, /*OP_CHECK_POS_END*/ Arguments.NON, /*OP_CHECK_LOOK_BEHIND_END*/ + Arguments.NON, /*OP_GRAPHEME_BOUNDARY*/ + Arguments.NON, /*OP_NOT_GRAPHEME_BOUNDARY*/ } : null; } diff --git a/third_party/joni/test/org/joni/test/TestPerlGraphemeBoundary.java b/third_party/joni/test/org/joni/test/TestPerlGraphemeBoundary.java new file mode 100644 index 0000000000..430934b4c5 --- /dev/null +++ b/third_party/joni/test/org/joni/test/TestPerlGraphemeBoundary.java @@ -0,0 +1,61 @@ +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni.test; + +import static org.junit.Assert.assertEquals; + +import java.nio.charset.StandardCharsets; + +import org.jcodings.specific.UTF8Encoding; +import org.joni.Option; +import org.joni.Regex; +import org.joni.Syntax; +import org.junit.Test; + +public class TestPerlGraphemeBoundary { + private static int search(String pattern, String input) { + byte[] patternBytes = pattern.getBytes(StandardCharsets.UTF_8); + byte[] inputBytes = input.getBytes(StandardCharsets.UTF_8); + Regex regex = new Regex(patternBytes, 0, patternBytes.length, Option.NONE, + UTF8Encoding.INSTANCE, Syntax.PerlNG); + return regex.matcher(inputBytes).search(0, inputBytes.length, Option.NONE); + } + + private static void assertBoundary(String left, String right) { + assertEquals(0, search("\\A" + left + "\\b{gcb}" + right + "\\z", left + right)); + } + + private static void assertNoBoundary(String left, String right) { + assertEquals(0, search("\\A" + left + "\\B{gcb}" + right + "\\z", left + right)); + } + + @Test + public void implementsUnicodeGraphemeBreakRules() { + assertEquals(0, search("\\A\\r\\B{gcb}\\n\\z", "\r\n")); + assertBoundary("A", "B"); + assertNoBoundary("ᄀ", "ᅡ"); + assertNoBoundary("A", "̈"); + assertNoBoundary("؀", "A"); + assertNoBoundary("क्", "क"); + assertNoBoundary("👩̈‍", "🚀"); + assertNoBoundary("🇦", "🇧"); + assertBoundary("🇦🇧", "🇨"); + } +} From 3d9c197f8a45a4d250d4a87037e6aa559ea34b66 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:18 +0200 Subject: [PATCH 21/60] refactor(regex): integrate native parser slices and retire Java rewrites (#1026) Squashed integration of PR #1026 at fb297a57b. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 35 +- docs/reference/feature-matrix.md | 4 +- .../runtime/regex/RegexPreprocessor.java | 398 +----------------- .../regex/lazy_negated_class_retirement.t | 34 ++ .../regex/omniholder_nullable_alternation.t | 42 ++ ...rminated_whitespace_quantifier_semantics.t | 25 ++ third_party/joni/src/org/joni/Lexer.java | 82 ++++ third_party/joni/src/org/joni/Parser.java | 58 +++ .../src/org/joni/exception/ErrorMessages.java | 13 + .../joni/test/org/joni/test/TestPerl.java | 25 ++ 10 files changed, 302 insertions(+), 414 deletions(-) create mode 100644 src/test/resources/unit/regex/lazy_negated_class_retirement.t create mode 100644 src/test/resources/unit/regex/omniholder_nullable_alternation.t create mode 100644 src/test/resources/unit/regex/terminated_whitespace_quantifier_semantics.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index e80850ad78..f74c5c668b 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,11 +224,13 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The current integration stack is preserved through review-ready PR #1024. It includes -the completed callback/runtime slices, lossless generated Unicode fixtures, -explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, -and the first 524 lines of retired Java-only preprocessor code. Every published -slice has a warning-free combined `make` checkpoint. +The published integration stack is preserved through draft PR #1025, stacked +on review-ready PR #1024. It includes the completed callback/runtime slices, +lossless generated Unicode fixtures, explicit `Is_*` property/value +normalization, fatal Joni syntax diagnostics, native GCB semantics, and the +first 524 lines of retired Java-only preprocessor code. Every published slice +has a warning-free combined `make` checkpoint. The current WIP integrates six +independently validated parser and preprocessor commits above that checkpoint. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -435,10 +437,13 @@ matcher-specific timeouts on both execution backends. - [x] Rejected 40 invalid Perl inline option/group-name forms in forked Joni with exact JVM/interpreter `reg_mesg.t` parity, reducing residual Joni-only acceptance differences from 198 to 158 (`028602adc`). - - [x] Validated native Python-style named captures and backreferences plus - removal of their frontend conversion (`afbe2bc34`). The 20-case oracle - passes on both execution backends with exact malformed/unknown diagnostics; - integration into the coordinator stack remains queued. + - [x] Integrated native Python-style named captures and backreferences plus + removal of their frontend conversion (`afbe2bc34`, integrated as + `cc489bee8`). The 20-case oracle passes on both execution backends with exact + malformed/unknown diagnostics. + - [x] Integrated native braced-octal parsing and missing-close/empty + diagnostics plus fatal unterminated braced-hex diagnostics (`55433291a`, + `913e2b583`) with exact JVM/interpreter `reg_mesg.t` parity. - [x] Fixed byte-mode substitution of upgraded marker regexes so chunks 05–10 exercise real boundary subjects with exact JVM/interpreter plans. - [x] Implemented native Joni GCB assertions for GB1–GB13 and GB999 and aligned @@ -488,9 +493,10 @@ matcher-specific timeouts on both execution backends. ### Next Steps -1. Land review-ready PR #1024, publish the native-GCB slice as its own stacked - PR, then integrate the validated braced-octal, braced-hex, - terminated-whitespace, and lazy-negated-class slices as focused stacked PRs. +1. Land review-ready PR #1024 and draft PR #1025, then publish the current + integration branch containing native braced-octal/hex diagnostics, + Python-style named groups, and the whitespace, lazy-negated-class, and DBIx + omniholder preprocessor retirements. 2. Continue boundary semantics natively in Joni in measured order: sentence breaks next (0/6,460 in the remainder of chunk 05), followed by line and word breaks (0/224,890 in chunks 06–10). Use bundled Perl 5.44 Unicode data and @@ -500,9 +506,8 @@ matcher-specific timeouts on both execution backends. Script/Script_Extensions, Numeric_Value, Joining_Group, General_Category, break-property values, and Age/In/Present_In. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. - Integrate the validated Python-style named-group and braced-octal slices, - then close alpha assertion aliases and underscored numeric escapes with - focused standard-Perl gates. + Close alpha assertion aliases and underscored numeric escapes with focused + standard-Perl gates. 4. Rerun the forced-Joni 80-file corpus on JVM and interpreter from the combined head. Save complete JSON and logs, publish the missing differential report, and compare every file with both the Phase 0 result and PR 958 under the diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index 1eeea4410e..d40a7ac26e 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -384,9 +384,9 @@ my @copy = @{$z}; # ERROR - ✅ **`\K` assertion**: Keep left — in `s///`, text before `\K` is preserved; match variables reflect only the portion after `\K`. - ✅ **Preprocessor**: `\Q`, `\L`, `\U`, `\l`, `\u`, `\E` are preprocessed in regex. - ✅ **Overloading**: `qr` overloading is implemented. See also [overload pragma](#pragmas). -- ❌ **Python-style named groups**: `(?P...)` and `(?P=name)` are accepted by Perl but are not yet parsed by Joni. +- ✅ **Python-style named groups**: `(?P...)` and `(?P=name)` are parsed natively by Joni with Perl capture numbering, duplicate-name behavior, and malformed/unknown-name diagnostics. - ❌ **Alpha assertion aliases**: `(*pla:...)`, `(*plb:...)`, `(*nla:...)`, `(*nlb:...)`, and `(*atomic:...)` are not yet parsed by Joni. -- ❌ **Underscored numeric regex escapes**: Perl spellings such as `\x{0_0_4_1}` and `\o{0_0_1_0_1}` are not yet parsed by Joni. Valid braced octal and unconditional missing-close/empty diagnostics are implemented in the pending parser stack; three `use re 'strict'` non-octal cases remain deferred. +- ❌ **Underscored numeric regex escapes**: Perl spellings such as `\x{0_0_4_1}` and `\o{0_0_1_0_1}` still rely on frontend normalization rather than native Joni parsing. Valid braced octal and unconditional missing-close/empty diagnostics are native; three `use re 'strict'` non-octal cases remain deferred. - ✅ **Dynamically-scoped regex variables**: Provisional captures, `$^R`, `$^N`, match positions, and callback locals follow matcher paths and unwind on backtracking. - ✅ **Recursive and Dynamic Patterns**: `(?R)`, `(?0)`, and runtime `(??{ code })` execute through Joni. Dynamic expressions may return strings or `qr//` values, nested alternatives participate in outer backtracking without changing outer grouping or capture numbering, and callback and pure-pattern recursion have engine-owned depth ceilings. diff --git a/src/main/java/org/perlonjava/runtime/regex/RegexPreprocessor.java b/src/main/java/org/perlonjava/runtime/regex/RegexPreprocessor.java index 5997a0189c..a901ac9963 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RegexPreprocessor.java +++ b/src/main/java/org/perlonjava/runtime/regex/RegexPreprocessor.java @@ -150,12 +150,9 @@ private static String preProcessRegexInternal(String s, RegexFlags regexFlags) { duplicateNameCounter = 0; warningsOnUse.clear(); - s = convertPythonStyleGroups(s); s = transformSimpleConditionals(s); s = removeUnderscoresFromEscapes(s); s = normalizeQuantifiers(s); - s = optimizeTerminatedWhitespaceQuantifiers(s); - s = optimizeTerminatedLazyNegatedClasses(s); // Expand multi-character case folds when case-insensitive flag is set if (regexFlags.isCaseInsensitive() && !regexFlags.isAsciiStrict() @@ -165,153 +162,7 @@ private static String preProcessRegexInternal(String s, RegexFlags regexFlags) { StringBuilder sb = new StringBuilder(); handleRegex(s, 0, sb, regexFlags, false); - String result = sb.toString(); - return preferOmniHolderLiteralAlternation(result); - } - - private static String optimizeTerminatedWhitespaceQuantifiers(String pattern) { - StringBuilder result = new StringBuilder(pattern.length()); - int i = 0; - int len = pattern.length(); - boolean escaped = false; - - while (i < len) { - char ch = pattern.charAt(i); - - if (escaped) { - result.append(ch); - escaped = false; - i++; - continue; - } - - if (ch == '[' && !startsExtendedCharacterClass(pattern, i)) { - int classEnd = findRegularCharacterClassEnd(pattern, i); - if (classEnd > i) { - result.append(pattern, i, classEnd + 1); - i = classEnd + 1; - continue; - } - } - - if (ch == '\\' - && i + 3 < len - && pattern.charAt(i + 1) == 's' - && (pattern.charAt(i + 2) == '*' || pattern.charAt(i + 2) == '+')) { - int literalOffset = i + 3; - if (pattern.charAt(literalOffset) == '+') { - result.append(pattern, i, literalOffset + 1); - i = literalOffset + 1; - continue; - } - if (pattern.charAt(literalOffset) == '?') { - literalOffset++; - } - - LiteralToken nextLiteral = readLiteralToken(pattern, literalOffset); - if (nextLiteral != null && !isPerlWhitespaceCodePoint(nextLiteral.codePoint)) { - result.append("\\s").append(pattern.charAt(i + 2)).append('+'); - i = literalOffset; - continue; - } - } - - if (ch == '\\') { - result.append(ch); - escaped = true; - i++; - continue; - } - - result.append(ch); - i++; - } - - return result.toString(); - } - - private static boolean isPerlWhitespaceCodePoint(int codePoint) { - return codePoint == '\t' - || codePoint == '\n' - || codePoint == 0x000B - || codePoint == '\f' - || codePoint == '\r' - || codePoint == ' ' - || codePoint == 0x0085 - || codePoint == 0x00A0 - || codePoint == 0x1680 - || (codePoint >= 0x2000 && codePoint <= 0x200A) - || codePoint == 0x2028 - || codePoint == 0x2029 - || codePoint == 0x202F - || codePoint == 0x205F - || codePoint == 0x3000; - } - - /** - * Java's regex engine can recurse deeply for repeated lazy negated classes, - * for example {@code /'[^']+?'/} applied thousands of times. When the next - * token is a literal that the negated class explicitly excludes, laziness is - * irrelevant: the class can never consume the delimiter. Make that specific - * quantifier possessive so Java does not build a deep lazy-loop stack. - */ - private static String optimizeTerminatedLazyNegatedClasses(String pattern) { - StringBuilder result = new StringBuilder(pattern.length()); - int i = 0; - int len = pattern.length(); - boolean escaped = false; - - while (i < len) { - char ch = pattern.charAt(i); - - if (escaped) { - result.append(ch); - escaped = false; - i++; - continue; - } - - if (ch == '\\') { - result.append(ch); - escaped = true; - i++; - continue; - } - - if (ch == '[' && !startsExtendedCharacterClass(pattern, i)) { - int classEnd = findRegularCharacterClassEnd(pattern, i); - if (classEnd > i) { - if (i + 1 < len - && pattern.charAt(i + 1) == '^' - && classEnd + 3 < len - && pattern.charAt(classEnd + 1) == '+' - && pattern.charAt(classEnd + 2) == '?') { - LiteralToken nextLiteral = readLiteralToken(pattern, classEnd + 3); - if (nextLiteral != null - && negatedClassExcludesLiteral(pattern, i, classEnd, nextLiteral.codePoint)) { - result.append(pattern, i, classEnd + 1); - result.append("++"); - i = classEnd + 3; - continue; - } - } - result.append(pattern, i, classEnd + 1); - i = classEnd + 1; - continue; - } - } - - result.append(ch); - i++; - } - - return result.toString(); - } - - private static boolean startsExtendedCharacterClass(String pattern, int bracketOffset) { - return bracketOffset >= 2 - && pattern.charAt(bracketOffset - 2) == '(' - && pattern.charAt(bracketOffset - 1) == '?'; + return sb.toString(); } private static int findRegularCharacterClassEnd(String pattern, int classStart) { @@ -353,239 +204,6 @@ private static int findRegularCharacterClassEnd(String pattern, int classStart) return -1; } - private static LiteralToken readLiteralToken(String pattern, int offset) { - if (offset >= pattern.length()) { - return null; - } - - int cp = pattern.codePointAt(offset); - if (cp == '\\') { - if (offset + 1 >= pattern.length()) { - return null; - } - return readEscapedLiteralToken(pattern, offset + 1); - } - - if (isRegexMetaCharacter(cp)) { - return null; - } - return new LiteralToken(cp, offset + Character.charCount(cp)); - } - - private static LiteralToken readEscapedLiteralToken(String pattern, int offset) { - int cp = pattern.codePointAt(offset); - int nextOffset = offset + Character.charCount(cp); - switch (cp) { - case 'n': - return new LiteralToken('\n', nextOffset); - case 'r': - return new LiteralToken('\r', nextOffset); - case 't': - return new LiteralToken('\t', nextOffset); - case 'f': - return new LiteralToken('\f', nextOffset); - case 'a': - return new LiteralToken(0x07, nextOffset); - case 'e': - return new LiteralToken(0x1B, nextOffset); - case 'c': - if (nextOffset < pattern.length()) { - int control = pattern.codePointAt(nextOffset); - return new LiteralToken(control ^ 64, nextOffset + Character.charCount(control)); - } - return null; - case 'x': - return readHexLiteralToken(pattern, nextOffset); - case 'o': - return readBracedRadixLiteralToken(pattern, nextOffset, 8); - default: - if (cp >= '0' && cp <= '7') { - return readOctalLiteralToken(pattern, offset); - } - if (isRegexEscapeWithVariableMeaning(cp)) { - return null; - } - return new LiteralToken(cp, nextOffset); - } - } - - private static LiteralToken readHexLiteralToken(String pattern, int offset) { - if (offset < pattern.length() && pattern.charAt(offset) == '{') { - return readBracedRadixLiteralToken(pattern, offset, 16); - } - - int value = 0; - int digits = 0; - int pos = offset; - while (pos < pattern.length() && digits < 2) { - int digit = Character.digit(pattern.charAt(pos), 16); - if (digit < 0) { - break; - } - value = value * 16 + digit; - pos++; - digits++; - } - return digits > 0 ? new LiteralToken(value, pos) : null; - } - - private static LiteralToken readBracedRadixLiteralToken(String pattern, int offset, int radix) { - if (offset >= pattern.length() || pattern.charAt(offset) != '{') { - return null; - } - int close = pattern.indexOf('}', offset + 1); - if (close < 0) { - return null; - } - String digits = pattern.substring(offset + 1, close).trim().replace("_", ""); - if (digits.isEmpty()) { - return null; - } - int value = 0; - for (int i = 0; i < digits.length(); i++) { - int digit = Character.digit(digits.charAt(i), radix); - if (digit < 0) { - return null; - } - value = value * radix + digit; - } - return new LiteralToken(value, close + 1); - } - - private static LiteralToken readOctalLiteralToken(String pattern, int offset) { - int value = 0; - int digits = 0; - int pos = offset; - while (pos < pattern.length() && digits < 3) { - char ch = pattern.charAt(pos); - if (ch < '0' || ch > '7') { - break; - } - value = value * 8 + (ch - '0'); - pos++; - digits++; - } - return digits > 0 ? new LiteralToken(value, pos) : null; - } - - private static boolean negatedClassExcludesLiteral(String pattern, int classStart, int classEnd, int literal) { - int pos = classStart + 2; // skip [^ - LiteralToken previous = null; - - while (pos < classEnd) { - if (pattern.charAt(pos) == '[' && pos + 1 < classEnd && pattern.charAt(pos + 1) == ':') { - return false; - } - - LiteralToken current; - if (pattern.charAt(pos) == '\\') { - if (pos + 1 >= classEnd) { - return false; - } - current = readEscapedLiteralToken(pattern, pos + 1); - if (current == null || current.endOffset > classEnd) { - return false; - } - } else { - int cp = pattern.codePointAt(pos); - current = new LiteralToken(cp, pos + Character.charCount(cp)); - } - - if (current.codePoint == literal) { - return true; - } - - if (previous != null - && previous.endOffset == pos - 1 - && pattern.charAt(pos - 1) == '-' - && previous.codePoint <= literal - && literal <= current.codePoint) { - return true; - } - - previous = current; - pos = current.endOffset; - - if (pos < classEnd && pattern.charAt(pos) == '-') { - pos++; - } - } - - return false; - } - - private static boolean isRegexMetaCharacter(int cp) { - return cp == '.' || cp == '^' || cp == '$' || cp == '|' || cp == '(' || cp == ')' - || cp == '[' || cp == ']' || cp == '{' || cp == '}' || cp == '*' - || cp == '+' || cp == '?'; - } - - private static boolean isRegexEscapeWithVariableMeaning(int cp) { - return cp == 'A' || cp == 'B' || cp == 'C' || cp == 'D' || cp == 'G' - || cp == 'H' || cp == 'K' || cp == 'N' || cp == 'P' || cp == 'R' - || cp == 'S' || cp == 'V' || cp == 'W' || cp == 'X' || cp == 'Z' - || cp == 'b' || cp == 'd' || cp == 'g' || cp == 'h' || cp == 'k' - || cp == 'p' || cp == 's' || cp == 'v' || cp == 'w' || cp == 'z'; - } - - private static class LiteralToken { - final int codePoint; - final int endOffset; - - LiteralToken(int codePoint, int endOffset) { - this.codePoint = codePoint; - this.endOffset = endOffset; - } - } - - /** - * Perl prefers the longest matching alternative when several branches succeed at the same - * offset; {@link Pattern} tries alternatives strictly left-to-right. DBIx::Simple's omniholder - * substitution uses {@code ($quoted|\(\?\?\))}: the nullable quoted repetition matches the empty - * string everywhere, so Java never reaches the literal {@code (??)} branch and SQL stays - * unchanged — JDBC then rejects {@code (??)} as invalid SQL. - *

- * Detect the usual Java-preprocessed shape {@code ( (?FLAGS:(?:'…'|"…")*)|\(\?\?\) )} - * (single outer capturing group) and swap alternatives so the omniholder literal is tried - * first. - */ - static String preferOmniHolderLiteralAlternation(String javaPattern) { - final String omniBranch = "\\(\\?\\?\\)"; - final String needle = "|" + omniBranch; - if (javaPattern.length() < omniBranch.length() + 4 - || javaPattern.charAt(0) != '(' - || javaPattern.charAt(javaPattern.length() - 1) != ')') { - return javaPattern; - } - int pipeIdx = javaPattern.lastIndexOf(needle); - if (pipeIdx <= 1) { - return javaPattern; - } - int omniStart = pipeIdx + 1; - if (!javaPattern.startsWith(omniBranch, omniStart)) { - return javaPattern; - } - int closingParenIdx = omniStart + omniBranch.length(); - if (closingParenIdx != javaPattern.length() - 1 || javaPattern.charAt(closingParenIdx) != ')') { - return javaPattern; - } - String leftAlt = javaPattern.substring(1, pipeIdx); - // Narrow heuristic so arbitrary ( A | \(??\) ) patterns are untouched. - if (!looksLikeQuotedStarAlternate(leftAlt)) { - return javaPattern; - } - return "(" + omniBranch + "|" + leftAlt + ")"; - } - - /** Left branch produced from Perl {@code (?:'[^']*'|"[^"]*")*} plus inline (?FLAGS: … ). */ - private static boolean looksLikeQuotedStarAlternate(String leftAlt) { - if (!leftAlt.startsWith("(?")) { - return false; - } - // Typical preprocessing embeds both quote flavours inside a non-capturing cluster. - return leftAlt.contains("(?:'[^") && leftAlt.contains("|\"[^"); - } - /** * Expand characters with multi-character case folds into alternations. * For example: ß → (?:ß|ss|SS|Ss|sS) @@ -2527,20 +2145,6 @@ private static boolean isValidQuantifierAt(String s, int offset) { } - private static String convertPythonStyleGroups(String pattern) { - // Convert (?P...) to (?...) - pattern = pattern.replaceAll("\\(\\?P<", "(?<"); - - // Convert (?P=name) to \k - StringBuffer result = new StringBuffer(); - Matcher m = Pattern.compile("\\(\\?P=([^)]+)\\)").matcher(pattern); - while (m.find()) { - m.appendReplacement(result, "\\\\k<" + m.group(1) + ">"); - } - m.appendTail(result); - return result.toString(); - } - /** * Transform simple conditional patterns (?(N)yes|no) that can be converted to alternations. *

diff --git a/src/test/resources/unit/regex/lazy_negated_class_retirement.t b/src/test/resources/unit/regex/lazy_negated_class_retirement.t new file mode 100644 index 0000000000..18af59eec9 --- /dev/null +++ b/src/test/resources/unit/regex/lazy_negated_class_retirement.t @@ -0,0 +1,34 @@ +use strict; +use warnings; +use Test::More; + +my $quoted = q('first' 'second'); +ok($quoted =~ /'([^']+?)'/, + 'lazy negated class finds the leftmost quoted token'); +is($1, 'first', + 'lazy excluded-delimiter capture is the first payload'); +is($&, q('first'), + 'lazy excluded-delimiter full match is leftmost and shortest'); + +my $shared = 'abXcdX'; +ok($shared =~ /\A([^Z]+?)X/, + 'lazy class that can consume delimiter still matches'); +is($1, 'ab', + 'lazy class gives the first delimiter to the suffix'); +is($&, 'abX', + 'lazy result stops at the first delimiter'); + +is('[value]' =~ /\A\[([^\]]+?)\]\z/ ? $1 : undef, 'value', + 'escaped closing bracket works in class and terminator'); +is('' =~ /\A<([^\x{3e}]+?)\x{3e}\z/ ? $1 : undef, 'value', + 'hex-escaped delimiter works in class and terminator'); +is('' =~ /\A<([^=>]+?)>\z/ ? $1 : undef, 'value', + 'class range that excludes delimiter preserves capture'); + +my $long = q(') . ('a' x 20_000) . q('); +ok($long =~ /\A'([^']+?)'\z/, + 'long lazy negated class reaches its excluded terminator'); +is(length($1), 20_000, + 'long lazy capture is complete'); + +done_testing; diff --git a/src/test/resources/unit/regex/omniholder_nullable_alternation.t b/src/test/resources/unit/regex/omniholder_nullable_alternation.t new file mode 100644 index 0000000000..2ce5666f5f --- /dev/null +++ b/src/test/resources/unit/regex/omniholder_nullable_alternation.t @@ -0,0 +1,42 @@ +use strict; +use warnings; +use Test::More; +require './src/main/perl/lib/DBIx/Simple.pm'; + +my $quoted = qr/(?:'[^']*'|"[^"]*")*/; +my $sql = 'SELECT * FROM t WHERE id IN (??)'; + +my @binds = (10, 20, 30); +my $rewritten = $sql; +my $replacements = 0; +$rewritten =~ s[($quoted|\(\?\?\))] { + $1 eq '(??)' + ? do { ++$replacements; '(' . join(', ', ('?') x @binds) . ')' } + : $1 +}eg; +is($rewritten, 'SELECT * FROM t WHERE id IN (?, ?, ?)', + 'exact DBIx substitution expands the omniholder'); +is($replacements, 1, 'exact DBIx substitution expands it once'); + +my $quoted_sql = q{SELECT '(??)', (??)}; +$replacements = 0; +$quoted_sql =~ s[($quoted|\(\?\?\))] { + $1 eq '(??)' ? do { ++$replacements; '(?)' } : $1 +}eg; +is($quoted_sql, q{SELECT '(??)', (?)}, + 'exact substitution leaves quoted omniholder untouched'); +is($replacements, 1, 'only the unquoted omniholder is replaced'); + +my $db = bless { dbd => 'SQLite' }, 'DBIx::Simple'; + +my $dbix_sql = 'INSERT INTO t VALUES (??)'; +$db->_replace_omniholder(\$dbix_sql, [10, 20, 30]); +is($dbix_sql, 'INSERT INTO t VALUES (?, ?, ?)', + 'bundled DBIx::Simple expands an unquoted omniholder'); + +my $dbix_quoted_sql = q{INSERT INTO t VALUES ('(??)', (??))}; +$db->_replace_omniholder(\$dbix_quoted_sql, [10, 20]); +is($dbix_quoted_sql, q{INSERT INTO t VALUES ('(??)', (?, ?))}, + 'bundled DBIx::Simple protects quoted text and expands the later omniholder'); + +done_testing; diff --git a/src/test/resources/unit/regex/terminated_whitespace_quantifier_semantics.t b/src/test/resources/unit/regex/terminated_whitespace_quantifier_semantics.t new file mode 100644 index 0000000000..e299f17931 --- /dev/null +++ b/src/test/resources/unit/regex/terminated_whitespace_quantifier_semantics.t @@ -0,0 +1,25 @@ +use strict; +use warnings; +use Test::More; + +my $two_spaces = " "; +ok($two_spaces =~ /\A\s*\x20\z/, + 'greedy whitespace star backtracks one literal space'); +ok($two_spaces =~ /\A\s+\x20\z/, + 'greedy whitespace plus backtracks one literal space'); + +my $two_tabs = "\t\t"; +ok($two_tabs =~ /\A\s*\t\z/, + 'greedy whitespace star backtracks one literal tab'); + +my $long = 'BEGIN' . (' ' x 20_000) . 'END'; +ok($long =~ /\ABEGIN\s+END\z/, + 'long greedy whitespace plus reaches its terminator'); +ok($long =~ /\ABEGIN\s*END\z/, + 'long greedy whitespace star reaches its terminator'); +ok($long =~ /\ABEGIN\s+?END\z/, + 'long lazy whitespace plus reaches its terminator'); +ok($long =~ /\ABEGIN\s*?END\z/, + 'long lazy whitespace star reaches its terminator'); + +done_testing; diff --git a/third_party/joni/src/org/joni/Lexer.java b/third_party/joni/src/org/joni/Lexer.java index 518261af31..4f0b5d883a 100644 --- a/third_party/joni/src/org/joni/Lexer.java +++ b/third_party/joni/src/org/joni/Lexer.java @@ -575,6 +575,7 @@ private void fetchTokenInCCFor_x() { int last = p; if (peekIs('{') && syntax.opEscXBraceHex8()) { + validatePerlBracedHexClose(); inc(); int num = scanUnsignedHexadecimalNumber(0, 8); if (num < 0) newValueException(ERR_TOO_BIG_WIDE_CHAR_VALUE); @@ -609,6 +610,80 @@ private void fetchTokenInCCFor_x() { } } + private void validatePerlBracedHexClose() { + if (!syntax.op2OptionPerl()) return; + + int cursor = p; + while (cursor < stop) { + int code = enc.mbcToCode(bytes, cursor, stop); + if (code == '}') return; + cursor += enc.length(bytes, cursor, stop); + } + newSyntaxException(PERL_MISSING_RIGHT_BRACE_ON_HEX_ESCAPE); + } + + private void fetchTokenFor_oBrace() { + int afterO = p; + if (!syntax.op2OptionPerl() || !left() || !peekIs('{')) { + c = env.convertBackslashValue('o'); + token.setC(c); + return; + } + + inc(); + int cursor = p; + int close = -1; + while (cursor < stop) { + int code = enc.mbcToCode(bytes, cursor, stop); + if (code == '}') { + close = cursor; + break; + } + cursor += enc.length(bytes, cursor, stop); + } + if (close < 0) { + newSyntaxException(PERL_MISSING_RIGHT_BRACE_ON_OCTAL_ESCAPE); + } + + boolean sawDigit = false; + boolean trailingWhitespace = false; + boolean invalid = false; + int value = 0; + cursor = p; + while (cursor < close) { + int code = enc.mbcToCode(bytes, cursor, close); + if (code == ' ' || code == '\t' || code == '\n' + || code == '\r' || code == '\f') { + if (sawDigit) trailingWhitespace = true; + } else if (code >= '0' && code <= '7' && !trailingWhitespace) { + sawDigit = true; + if (value > (Integer.MAX_VALUE - (code - '0')) / 8) { + newValueException(TOO_BIG_NUMBER); + } + value = value * 8 + code - '0'; + } else { + invalid = true; + } + cursor += enc.length(bytes, cursor, close); + } + + if (invalid) { + p = afterO; + c = env.convertBackslashValue('o'); + token.setC(c); + return; + } + if (!sawDigit) { + newSyntaxException(PERL_EMPTY_OCTAL_ESCAPE); + } + + p = close; + inc(); + token.type = TokenType.CODE_POINT; + token.base = 8; + token.setCode(value); + } + private void fetchTokenInCCFor_u() { if (!left()) return; int last = p; @@ -734,6 +809,9 @@ protected final TokenType fetchTokenInCC() { case 'x': fetchTokenInCCFor_x(); break; + case 'o': + fetchTokenFor_oBrace(); + break; case 'u': fetchTokenInCCFor_u(); break; @@ -803,6 +881,7 @@ private void fetchTokenFor_xBrace() { int last = p; if (peekIs('{') && syntax.opEscXBraceHex8()) { + validatePerlBracedHexClose(); inc(); int num = scanUnsignedHexadecimalNumber(0, 8); if (num < 0) newValueException(ERR_TOO_BIG_WIDE_CHAR_VALUE); @@ -1193,6 +1272,9 @@ protected final void fetchToken() { case 'x': fetchTokenFor_xBrace(); break; + case 'o': + fetchTokenFor_oBrace(); + break; case 'u': fetchTokenFor_uHex(); break; diff --git a/third_party/joni/src/org/joni/Parser.java b/third_party/joni/src/org/joni/Parser.java index 40971c1cfb..5eeac74765 100644 --- a/third_party/joni/src/org/joni/Parser.java +++ b/third_party/joni/src/org/joni/Parser.java @@ -56,6 +56,7 @@ import org.joni.constants.internal.NodeType; import org.joni.constants.internal.TokenType; import org.joni.exception.ErrorMessages; +import org.joni.exception.SyntaxException; class Parser extends Lexer { protected int returnCode; // return code used by parser methods (they itself return parsed nodes) @@ -527,6 +528,53 @@ private Node parseEnclose(TokenType term) { } break; + case 'P': /* Perl's Python-style (?P...) and (?P=name) */ + if (!syntax.op2OptionPerl() || !left()) { + newSyntaxException(UNDEFINED_GROUP_OPTION); + } + fetch(); + if (c == '<') { + if (!left()) { + newSyntaxException(PERL_PYTHON_NAMED_CAPTURE_NOT_TERMINATED); + } + int first = enc.mbcToCode(bytes, p, stop); + if (enc.isDigit(first) || !enc.isWord(first)) { + newValueException(PERL_GROUP_NAME_MUST_START_WITH_WORD); + } + if (!hasCodePointAhead('>')) { + newSyntaxException(PERL_PYTHON_NAMED_CAPTURE_NOT_TERMINATED); + } + listCapture = false; + node = parseEncloseNamedGroup2(listCapture); + break; + } + if (c == '=') { + if (!left()) { + newSyntaxException(PERL_PYTHON_NAMED_BACKREF_NOT_TERMINATED); + } + int first = enc.mbcToCode(bytes, p, stop); + if (enc.isDigit(first) || !enc.isWord(first)) { + newValueException(PERL_GROUP_NAME_MUST_START_WITH_WORD); + } + if (!hasCodePointAhead(')')) { + newSyntaxException(PERL_PYTHON_NAMED_BACKREF_NOT_TERMINATED); + } + c = '('; + try { + fetchNamedBackrefToken(); + } catch (SyntaxException e) { + if (e.getMessage().startsWith("undefined name <")) { + newValueException(PERL_REFERENCE_TO_NONEXISTENT_NAMED_GROUP); + } + throw e; + } + returnCode = 0; + return parseBackref(); + } + newValueException(PERL_PYTHON_GROUP_SEQUENCE_NOT_RECOGNIZED, + new String(Character.toChars(c))); + break; + case '(': /* conditional expression: (?(cond)yes), (?(cond)yes|no) */ if (left() && syntax.op2QMarkLParenCondition()) { int num = -1; @@ -956,6 +1004,16 @@ private Node parseEncloseNamedGroup2(boolean listCapture) { return en; } + private boolean hasCodePointAhead(int target) { + int cursor = p; + while (cursor < stop) { + int code = enc.mbcToCode(bytes, cursor, stop); + if (code == target) return true; + cursor += enc.length(bytes, cursor, stop); + } + return false; + } + private int findStrPosition(int[]s, int n, int from, int to, Ptr nextChar) { int x; int q; diff --git a/third_party/joni/src/org/joni/exception/ErrorMessages.java b/third_party/joni/src/org/joni/exception/ErrorMessages.java index d36a50c32b..45d6589337 100644 --- a/third_party/joni/src/org/joni/exception/ErrorMessages.java +++ b/third_party/joni/src/org/joni/exception/ErrorMessages.java @@ -59,6 +59,14 @@ public interface ErrorMessages extends org.jcodings.exception.ErrorMessages { String INVALID_CONDITION_PATTERN = "invalid conditional pattern"; String PERL_GROUP_NAME_MUST_START_WITH_WORD = "Group name must start with a non-digit word character"; + String PERL_REFERENCE_TO_NONEXISTENT_NAMED_GROUP = + "Reference to nonexistent named group"; + String PERL_PYTHON_GROUP_SEQUENCE_NOT_RECOGNIZED = + "Sequence (?P%n...) not recognized"; + String PERL_PYTHON_NAMED_CAPTURE_NOT_TERMINATED = + "Sequence (?<... not terminated"; + String PERL_PYTHON_NAMED_BACKREF_NOT_TERMINATED = + "Sequence ?P=... not terminated"; String PERL_CARET_MINUS_OPTION_NOT_RECOGNIZED = "Sequence (?^-...) not recognized"; String PERL_CARET_D_OPTION_NOT_RECOGNIZED = @@ -71,6 +79,11 @@ public interface ErrorMessages extends org.jcodings.exception.ErrorMessages { "Regexp modifier \"l\" may not appear twice"; String PERL_MODIFIER_A_MAXIMUM_TWICE = "Regexp modifier \"a\" may appear a maximum of twice"; + String PERL_MISSING_RIGHT_BRACE_ON_OCTAL_ESCAPE = + "Missing right brace on \\o{}"; + String PERL_EMPTY_OCTAL_ESCAPE = "Empty \\o{}"; + String PERL_MISSING_RIGHT_BRACE_ON_HEX_ESCAPE = + "Missing right brace on \\x{}"; /* values error (syntax error) */ String TOO_BIG_NUMBER = "too big number"; diff --git a/third_party/joni/test/org/joni/test/TestPerl.java b/third_party/joni/test/org/joni/test/TestPerl.java index a4fb6f6bf6..0ad090cde9 100755 --- a/third_party/joni/test/org/joni/test/TestPerl.java +++ b/third_party/joni/test/org/joni/test/TestPerl.java @@ -51,6 +51,31 @@ public void test() throws Exception { xerrs("(?da:foo)", ErrorMessages.PERL_MODIFIERS_D_AND_A_MUTUALLY_EXCLUSIVE); xerrs("(?lil:foo)", ErrorMessages.PERL_MODIFIER_L_MAY_NOT_APPEAR_TWICE); xerrs("(?aaia:foo)", ErrorMessages.PERL_MODIFIER_A_MAXIMUM_TWICE); + xerrs("\\o{7", ErrorMessages.PERL_MISSING_RIGHT_BRACE_ON_OCTAL_ESCAPE); + xerrs("[\\o{7]", ErrorMessages.PERL_MISSING_RIGHT_BRACE_ON_OCTAL_ESCAPE); + xerrs("\\o{}", ErrorMessages.PERL_EMPTY_OCTAL_ESCAPE); + xerrs("[\\o{}]", ErrorMessages.PERL_EMPTY_OCTAL_ESCAPE); + x2s("\\o{141}", "a", 0, 1); + x2s("\\o{ 141 }", "a", 0, 1); + ns("\\o{789}", ""); + xerrs("\\x{X", ErrorMessages.PERL_MISSING_RIGHT_BRACE_ON_HEX_ESCAPE); + xerrs("[\\x{X]", ErrorMessages.PERL_MISSING_RIGHT_BRACE_ON_HEX_ESCAPE); + x2s("\\x{61}", "a", 0, 1); + ns("\\x{}", ""); + ns("\\x{X}", ""); + x2s("(?Pa)(?P=word)", "aa", 0, 2); + ns("(?Pa)(?P=word)", "ab"); + x2s("(?:(?Pa)|(?Pb))(?P=x)", "aa", 0, 2); + x2s("(?:(?Pa)|(?Pb))(?P=x)", "bb", 0, 2); + ns("(?:(?Pa)|(?Pb))(?P=x)", "ab"); + xerrs("(?P=missing)", ErrorMessages.PERL_REFERENCE_TO_NONEXISTENT_NAMED_GROUP); + xerrs("(?P<)", ErrorMessages.PERL_GROUP_NAME_MUST_START_WITH_WORD); + xerrs("(?P=)", ErrorMessages.PERL_GROUP_NAME_MUST_START_WITH_WORD); + xerrs("(?PXfoo)", "Sequence (?PX...) not recognized"); + xerrs("(?P)]", "z"); + x2s("\\(\\?P", "(?P", 0, 6); } @org.junit.Test(timeout = 5000) From e01027576ffa72a011470e2b7a3678c05d0ee508 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:19 +0200 Subject: [PATCH 22/60] feat(regex): implement native Perl sentence boundaries (#1027) Squashed integration of PR #1027 at bf77f092c. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 46 +- dev/tools/generate_joni_boundary_data.pl | 114 +++ docs/reference/feature-matrix.md | 3 +- .../unit/regex/sentence_boundary_rules.t | 48 ++ .../joni/src/org/joni/ArrayCompiler.java | 8 + .../joni/src/org/joni/ByteCodeMachine.java | 131 ++++ third_party/joni/src/org/joni/Lexer.java | 12 +- .../joni/src/org/joni/SentenceBreakData.java | 681 ++++++++++++++++++ .../joni/constants/internal/AnchorType.java | 6 + .../org/joni/constants/internal/OPCode.java | 6 + .../joni/test/TestPerlSentenceBoundary.java | 65 ++ 11 files changed, 1100 insertions(+), 20 deletions(-) create mode 100644 dev/tools/generate_joni_boundary_data.pl create mode 100644 src/test/resources/unit/regex/sentence_boundary_rules.t create mode 100644 third_party/joni/src/org/joni/SentenceBreakData.java create mode 100644 third_party/joni/test/org/joni/test/TestPerlSentenceBoundary.java diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index f74c5c668b..386164b01c 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,13 +224,15 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1025, stacked -on review-ready PR #1024. It includes the completed callback/runtime slices, +The published integration stack is preserved through draft PR #1026, stacked +on draft PR #1025 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the first 524 lines of retired Java-only preprocessor code. Every published slice -has a warning-free combined `make` checkpoint. The current WIP integrates six -independently validated parser and preprocessor commits above that checkpoint. +has a warning-free combined `make` checkpoint. The current WIP adds native +sentence boundaries above that checkpoint; independently validated alpha +assertion aliases and global zero-width `/g` progression remain parallel +integration slices. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -250,9 +252,17 @@ Authoritative chunk 05 improves by 6,324 assertions from 2,192/14,953 to 8,516/14,976 identically on JVM and interpreter: its complete GCB/`\X` section passes, leaving only the 6,460 sentence-boundary assertions in that chunk. +Native Joni sentence assertions now implement SB1–SB11 and SB998 with a +reproducibly generated Perl 5.44 Unicode 17.0 `Sentence_Break` table. The +23-assertion focused oracle passes on system Perl, JVM, and interpreter, direct +Joni coverage exercises the same engine path, and authoritative chunk 05 passes +14,976/14,976 identically on JVM and interpreter. This closes all 6,460 +remaining sentence assertions without coupling the Joni fork to ICU or the +PerlOnJava runtime. + The most recent exact property chunks 01–04 remain 98,092/167,501 on both -execution backends. Combined with the native-GCB result and unchanged chunks -06–10, current generated evidence is 106,608/407,367. A resource-contended +execution backends. Combined with the complete boundary chunk 05 and unchanged +chunks 06–10, current generated evidence is 113,068/407,367. A resource-contended current-head refresh did not reproduce a complete exact JVM/interpreter pair, so it does not replace that accepted baseline. @@ -449,7 +459,10 @@ matcher-specific timeouts on both execution backends. - [x] Implemented native Joni GCB assertions for GB1–GB13 and GB999 and aligned `\X` with repeated GB9c Indic conjunct behavior. The focused oracle passes 29/29 and generated chunk 05 reaches 8,516/14,976 on both execution backends. - - [ ] Implement native Joni line, sentence, and word boundary algorithms, then + - [x] Implemented native Joni sentence assertions for SB1–SB11 and SB998 from + a reproducible Perl 5.44 Unicode 17.0 table. The focused oracle passes 23/23 + and generated chunk 05 passes 14,976/14,976 on both execution backends. + - [ ] Implement native Joni line and word boundary algorithms, then close the remaining property and boundary failures before marking Phase 3 complete. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate @@ -493,21 +506,22 @@ matcher-specific timeouts on both execution backends. ### Next Steps -1. Land review-ready PR #1024 and draft PR #1025, then publish the current - integration branch containing native braced-octal/hex diagnostics, - Python-style named groups, and the whitespace, lazy-negated-class, and DBIx - omniholder preprocessor retirements. -2. Continue boundary semantics natively in Joni in measured order: sentence - breaks next (0/6,460 in the remainder of chunk 05), followed by line and word - breaks (0/224,890 in chunks 06–10). Use bundled Perl 5.44 Unicode data and +1. Land review-ready PR #1024 and draft PRs #1025–#1026, then publish the + sentence-boundary slice as the next stacked PR. Integrate the independently + validated alpha-assertion alias commit and the pending zero-width `/g` + adapter commit after their non-overlapping gates are complete. +2. Continue boundary semantics natively in Joni with line and word breaks + (0/224,890 in chunks 06–10). PerlOnJava4 owns line-boundary preparation while + the coordinator takes word boundaries after the shared sentence base is + published. Use pinned Perl 5.44 Unicode data and the generated chunks as the release oracle; remove each simplified Java boundary rewrite only after native parity. 3. Implement the remaining property clusters in measured order: Block, Script/Script_Extensions, Numeric_Value, Joining_Group, General_Category, break-property values, and Age/In/Present_In. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. - Close alpha assertion aliases and underscored numeric escapes with focused - standard-Perl gates. + Integrate the completed alpha assertion aliases and close underscored numeric + escapes with focused standard-Perl gates. 4. Rerun the forced-Joni 80-file corpus on JVM and interpreter from the combined head. Save complete JSON and logs, publish the missing differential report, and compare every file with both the Phase 0 result and PR 958 under the diff --git a/dev/tools/generate_joni_boundary_data.pl b/dev/tools/generate_joni_boundary_data.pl new file mode 100644 index 0000000000..9cf1c38d3f --- /dev/null +++ b/dev/tools/generate_joni_boundary_data.pl @@ -0,0 +1,114 @@ +#!/usr/bin/env perl +use strict; +use warnings; + +my $unicode_root = 'perl5/lib/unicore'; +my $expected_unicode_version = '17.0.0'; +open my $version_fh, '<', "$unicode_root/version" or die "Can't read Unicode version: $!\n"; +chomp(my $unicode_version = <$version_fh>); +close $version_fh; +die "Expected Unicode $expected_unicode_version, found $unicode_version\n" + unless $unicode_version eq $expected_unicode_version; + +open my $property_fh, '<', "$unicode_root/auxiliary/SentenceBreakProperty.txt" + or die "Can't read Sentence_Break data: $!\n"; +my @ranges; +while (<$property_fh>) { + next unless /^([0-9A-F]+)(?:\.\.([0-9A-F]+))?\s*;\s*([A-Za-z_]+)/; + push @ranges, [hex($1), defined($2) ? hex($2) : hex($1), $3]; +} +close $property_fh; +@ranges = sort { $a->[0] <=> $b->[0] } @ranges; + +my (@starts, @values); +my $cursor = 0; +for my $range (@ranges) { + my ($start, $end, $property) = @$range; + die "Overlapping Sentence_Break ranges at U+", sprintf('%04X', $start), "\n" + if $start < $cursor; + if ($start > $cursor) { + push @starts, $cursor; + push @values, 'Other'; + } + push @starts, $start; + push @values, $property; + $cursor = $end + 1; +} +if ($cursor <= 0x10ffff) { + push @starts, $cursor; + push @values, 'Other'; +} + +my @classes = qw( + Other ATerm Close CR Extend Format LF Lower Numeric OLetter SContinue Sep Sp STerm Upper +); +my %class_id; +@class_id{@classes} = (0 .. $#classes); + +for my $value (@values) { + die "Unknown Sentence_Break value '$value'\n" unless exists $class_id{$value}; +} + +print <<'HEADER'; +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni; + +// Generated by dev/tools/generate_joni_boundary_data.pl. Do not edit manually. +final class SentenceBreakData { +HEADER + +print " static final String UNICODE_VERSION = \"$unicode_version\";\n"; +for my $i (0 .. $#classes) { + my $constant = uc($classes[$i]); + print " static final byte $constant = $i;\n"; +} + +print "\n private static final int[] STARTS = {\n"; +for (my $i = 0; $i < @starts; $i += 10) { + my $end = $i + 9 < $#starts ? $i + 9 : $#starts; + print " ", join(', ', map { sprintf '0x%X', $starts[$_] } $i .. $end), ",\n"; +} +print " };\n\n private static final byte[] VALUES = {\n"; +for (my $i = 0; $i < @values; $i += 16) { + my $end = $i + 15 < $#values ? $i + 15 : $#values; + print " ", join(', ', map { uc($values[$_]) } $i .. $end), ",\n"; +} +print <<'FOOTER'; + }; + + static byte propertyOf(int codePoint) { + int low = 0; + int high = STARTS.length - 1; + while (low <= high) { + int middle = (low + high) >>> 1; + if (STARTS[middle] <= codePoint) { + low = middle + 1; + } else { + high = middle - 1; + } + } + return VALUES[Math.max(0, high)]; + } + + private SentenceBreakData() { + } +} +FOOTER diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index d40a7ac26e..d0c3f10f94 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -397,8 +397,9 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 106,608/407,367 identically on JVM and interpreter: chunks 01–04 expose property/value alias gaps, and the remaining boundary failures are sentence, line, and word breaks. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 113,068/407,367 identically on JVM and interpreter: chunks 01–04 expose property/value alias gaps, and the remaining boundary failures are line and word breaks. - ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. +- ✅ **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter. - ✅ **Embedded Code in Regex**: `(?{ code })`, optimistic callbacks `(*{ code })`, executable callback conditions, and `(??{ code })` run as lexical closures in Joni with provisional captures and backtracking unwind. Callback `local` frames follow matcher paths, and escaped loop control or `goto` stops at the callback pseudo-block boundary. `$^N` follows capture-close order independently of `$+`. - ✅ **Regex Debugging**: Lexically scoped `use/no re 'debug'` and `debugcolor` are supported, including runtime snapshot ownership. - ✅ **Runtime Regex Evaluation**: `use re 'eval'` controls whether interpolated patterns containing eval groups may compile. Admitted runtime source is compiled into lexical callback closures and preserves its package, visible lexical cells, Unicode or byte source type, default regex modifiers, and match-once state. Literal callbacks, interpolated `qr//` values (including local, referenced, and tied arrays), raw runtime eval groups, callback conditions, and standalone `(?(DEFINE)...)` containers may be composed in one Joni pattern; dynamic callbacks may return further admitted executable source. diff --git a/src/test/resources/unit/regex/sentence_boundary_rules.t b/src/test/resources/unit/regex/sentence_boundary_rules.t new file mode 100644 index 0000000000..5b54af34cd --- /dev/null +++ b/src/test/resources/unit/regex/sentence_boundary_rules.t @@ -0,0 +1,48 @@ +use strict; +use warnings; +use utf8; +use Test::More tests => 23; + +sub boundary_at { + my ($subject, $offset) = @_; + my $left = quotemeta substr($subject, 0, $offset); + my $right = quotemeta substr($subject, $offset); + return $subject =~ /\A${left}\b{sb}${right}\z/; +} + +sub no_boundary_at { + my ($subject, $offset) = @_; + my $left = quotemeta substr($subject, 0, $offset); + my $right = quotemeta substr($subject, $offset); + return $subject =~ /\A${left}\B{sb}${right}\z/; +} + +ok(boundary_at('ab', 0), 'SB1 start of text is a boundary'); +ok(boundary_at('ab', 2), 'SB2 end of text is a boundary'); +ok(no_boundary_at("\r\n", 1), 'SB3 keeps CR LF together'); +ok(boundary_at("\rA", 1), 'SB4 breaks after CR'); +ok(no_boundary_at("A\x{0308}", 1), 'SB5 ignores Extend before sentence breaking'); +ok(no_boundary_at("A\x{200C}", 1), 'SB5 ignores Format before sentence breaking'); +ok(no_boundary_at('.1', 1), 'SB6 keeps ATerm before Numeric'); +ok(no_boundary_at('A.B', 2), 'SB7 keeps an initial before Upper'); +ok(no_boundary_at('A. a', 3), 'SB8 keeps ATerm and spaces before Lower'); +ok(no_boundary_at('A.) a', 4), 'SB8 skips Close and Sp before Lower'); +ok(no_boundary_at("etc.)\x{2019}\x{a0}\x{2018}(the", 7), + 'SB8 looks through trailing Close characters before Lower'); +ok(no_boundary_at('A. ,', 3), 'SB8a keeps sentence continuation punctuation'); +ok(no_boundary_at('A.)', 2), 'SB9 keeps Close after a terminal'); +ok(no_boundary_at('A. ', 3), 'SB10 keeps consecutive spaces after a terminal'); +ok(boundary_at('A. B', 3), 'SB11 breaks after ATerm and Sp'); +ok(boundary_at('A!B', 2), 'SB11 breaks after STerm'); +ok(boundary_at('A!a', 2), 'SB8 does not suppress STerm before Lower'); +ok(boundary_at("A!\x{0308}B", 3), 'ignored characters preserve terminal context'); +ok(no_boundary_at("A.\n\x{0308}B", 4), + 'ignored characters after a paragraph break do not reopen SB11 context'); +ok(no_boundary_at('AB', 1), 'SB998 keeps ordinary text in one sentence'); +ok(!boundary_at('AB', 1), 'positive assertion rejects a non-boundary'); +ok(!no_boundary_at('A!B', 2), 'negated assertion rejects a sentence boundary'); + +{ + use bytes; + ok('A!B' =~ /\AA!\b{sb}B\z/, 'byte-mode ASCII uses sentence boundaries'); +} diff --git a/third_party/joni/src/org/joni/ArrayCompiler.java b/third_party/joni/src/org/joni/ArrayCompiler.java index 7a1ffe5e7a..fd66c8e15f 100644 --- a/third_party/joni/src/org/joni/ArrayCompiler.java +++ b/third_party/joni/src/org/joni/ArrayCompiler.java @@ -1206,6 +1206,14 @@ protected void compileAnchorNode(AnchorNode node) { addOpcode(OPCode.NOT_GRAPHEME_BOUNDARY); break; + case AnchorType.SENTENCE_BOUNDARY: + addOpcode(OPCode.SENTENCE_BOUNDARY); + break; + + case AnchorType.NOT_SENTENCE_BOUNDARY: + addOpcode(OPCode.NOT_SENTENCE_BOUNDARY); + break; + case AnchorType.WORD_BEGIN: if (Config.USE_WORD_BEGIN_END) { if (node.asciiRange) { diff --git a/third_party/joni/src/org/joni/ByteCodeMachine.java b/third_party/joni/src/org/joni/ByteCodeMachine.java index 88739ce6a2..36897666c6 100644 --- a/third_party/joni/src/org/joni/ByteCodeMachine.java +++ b/third_party/joni/src/org/joni/ByteCodeMachine.java @@ -252,6 +252,8 @@ private final int execute(final boolean checkThreadInterrupt) throws Interrupted case OPCode.NOT_WORD_BOUND: opNotWordBound(); continue; case OPCode.GRAPHEME_BOUNDARY: opGraphemeBoundary(false); continue; case OPCode.NOT_GRAPHEME_BOUNDARY: opGraphemeBoundary(true); continue; + case OPCode.SENTENCE_BOUNDARY: opSentenceBoundary(false); continue; + case OPCode.NOT_SENTENCE_BOUNDARY: opSentenceBoundary(true); continue; case OPCode.WORD_BEGIN: opWordBegin(); continue; case OPCode.WORD_END: opWordEnd(); continue; @@ -411,6 +413,8 @@ private final int executeSb(final boolean checkThreadInterrupt) throws Interrupt case OPCode.NOT_WORD_BOUND: opNotWordBoundSb(); continue; case OPCode.GRAPHEME_BOUNDARY: opGraphemeBoundary(false); continue; case OPCode.NOT_GRAPHEME_BOUNDARY: opGraphemeBoundary(true); continue; + case OPCode.SENTENCE_BOUNDARY: opSentenceBoundary(false); continue; + case OPCode.NOT_SENTENCE_BOUNDARY: opSentenceBoundary(true); continue; case OPCode.WORD_BEGIN: opWordBeginSb(); continue; case OPCode.WORD_END: opWordEndSb(); continue; @@ -1247,6 +1251,133 @@ private void opGraphemeBoundary(boolean negated) { if (isGraphemeBoundary() == negated) opFail(); } + private void opSentenceBoundary(boolean negated) { + if (isSentenceBoundary() == negated) opFail(); + } + + private boolean isSentenceBoundary() { + if (s <= str || s >= end) return true; // SB1, SB2 + + int leftPosition = enc.prevCharHead(bytes, str, s, end); + byte left = sentencePropertyAt(leftPosition); + byte right = sentencePropertyAt(s); + + if (left == SentenceBreakData.CR && right == SentenceBreakData.LF) return false; // SB3 + if (isSentenceParagraphSeparator(left)) return true; // SB4 + if (isSentenceIgnored(right)) return false; // SB5 + + int[] position = {leftPosition}; + byte previous = previousSentenceProperty(position); + + if (previous == SentenceBreakData.ATERM && right == SentenceBreakData.NUMERIC) return false; // SB6 + if (right == SentenceBreakData.UPPER && previous == SentenceBreakData.ATERM) { + byte beforeTerm = previousSentenceProperty(position); + if (beforeTerm == SentenceBreakData.UPPER || beforeTerm == SentenceBreakData.LOWER) return false; // SB7 + } + + if (sentenceATermBeforeCloseAndSpace(positionFor(leftPosition)) + && sentenceLowerAhead(s)) return false; // SB8 + + if ((right == SentenceBreakData.SCONTINUE + || right == SentenceBreakData.STERM + || right == SentenceBreakData.ATERM) + && sentenceTerminalBefore(positionFor(leftPosition), true)) return false; // SB8a + + if ((right == SentenceBreakData.CLOSE + || right == SentenceBreakData.SP + || isSentenceParagraphSeparator(right)) + && sentenceTerminalBeforeClose(positionFor(leftPosition))) return false; // SB9 + + if ((right == SentenceBreakData.SP || isSentenceParagraphSeparator(right)) + && sentenceTerminalBefore(positionFor(leftPosition), true)) return false; // SB10 + + // Extend and Format are ignored except after a paragraph separator. A + // separator followed by ignored characters has already broken at SB4; + // do not let SB11 scan back through it to an earlier terminal. + if (isSentenceIgnored(left) && isSentenceParagraphSeparator(previous)) return false; + + if (sentenceBoundaryAfterTerminal(positionFor(leftPosition))) return true; // SB11 + + return false; // SB998 + } + + private int[] positionFor(int position) { + return new int[] {position}; + } + + private byte sentencePropertyAt(int position) { + return SentenceBreakData.propertyOf(enc.mbcToCode(bytes, position, end)); + } + + private byte previousSentenceProperty(int[] position) { + while (position[0] >= str) { + byte property = sentencePropertyAt(position[0]); + if (position[0] == str) { + position[0] = -1; + } else { + position[0] = enc.prevCharHead(bytes, str, position[0], end); + } + if (!isSentenceIgnored(property)) return property; + } + return SentenceBreakData.OTHER; + } + + private boolean sentenceTerminalBefore(int[] position, boolean skipSpaces) { + byte property = previousSentenceProperty(position); + if (skipSpaces) { + while (property == SentenceBreakData.SP) property = previousSentenceProperty(position); + } + while (property == SentenceBreakData.CLOSE) property = previousSentenceProperty(position); + return isSentenceTerminal(property); + } + + private boolean sentenceATermBeforeCloseAndSpace(int[] position) { + byte property = previousSentenceProperty(position); + while (property == SentenceBreakData.SP) property = previousSentenceProperty(position); + while (property == SentenceBreakData.CLOSE) property = previousSentenceProperty(position); + return property == SentenceBreakData.ATERM; + } + + private boolean sentenceLowerAhead(int position) { + while (position < end) { + byte property = sentencePropertyAt(position); + if (property == SentenceBreakData.LOWER) return true; + if (property != SentenceBreakData.CLOSE + && property != SentenceBreakData.SP + && !isSentenceIgnored(property)) return false; + position += enc.length(bytes, position, end); + } + return false; + } + + private boolean sentenceTerminalBeforeClose(int[] position) { + byte property = previousSentenceProperty(position); + while (property == SentenceBreakData.CLOSE) property = previousSentenceProperty(position); + return isSentenceTerminal(property); + } + + private boolean sentenceBoundaryAfterTerminal(int[] position) { + byte property = previousSentenceProperty(position); + if (isSentenceParagraphSeparator(property)) property = previousSentenceProperty(position); + while (property == SentenceBreakData.SP) property = previousSentenceProperty(position); + while (property == SentenceBreakData.CLOSE) property = previousSentenceProperty(position); + return isSentenceTerminal(property); + } + + private boolean isSentenceIgnored(byte property) { + return property == SentenceBreakData.EXTEND || property == SentenceBreakData.FORMAT; + } + + private boolean isSentenceParagraphSeparator(byte property) { + return property == SentenceBreakData.SEP + || property == SentenceBreakData.CR + || property == SentenceBreakData.LF; + } + + private boolean isSentenceTerminal(byte property) { + return property == SentenceBreakData.ATERM || property == SentenceBreakData.STERM; + } + private boolean isGraphemeBoundary() { if (s <= str || s >= end) return true; // GB1, GB2 diff --git a/third_party/joni/src/org/joni/Lexer.java b/third_party/joni/src/org/joni/Lexer.java index 4f0b5d883a..b32d8413ef 100644 --- a/third_party/joni/src/org/joni/Lexer.java +++ b/third_party/joni/src/org/joni/Lexer.java @@ -1209,14 +1209,14 @@ protected final void fetchToken() { break; case 'b': if (syntax.opEscBWordBound()) { - if (fetchTokenForPerlGraphemeBoundary(false)) break; + if (fetchTokenForPerlBoundary(false)) break; fetchTokenFor_anchor(AnchorType.WORD_BOUND); token.setAnchorASCIIRange(isAsciiRange(env.option) && !isWordBoundAllRange(env.option)); } break; case 'B': if (syntax.opEscBWordBound()) { - if (fetchTokenForPerlGraphemeBoundary(true)) break; + if (fetchTokenForPerlBoundary(true)) break; fetchTokenFor_anchor(AnchorType.NOT_WORD_BOUND); token.setAnchorASCIIRange(isAsciiRange(env.option) && !isWordBoundAllRange(env.option)); } @@ -1427,7 +1427,7 @@ protected final void fetchToken() { } // while } - private boolean fetchTokenForPerlGraphemeBoundary(boolean negated) { + private boolean fetchTokenForPerlBoundary(boolean negated) { if (!syntax.op2OptionPerl() || !left() || peek() != '{') return false; mark(); @@ -1442,6 +1442,12 @@ private boolean fetchTokenForPerlGraphemeBoundary(boolean negated) { : AnchorType.GRAPHEME_BOUNDARY); return true; } + if (name.toString().equals("sb")) { + fetchTokenFor_anchor(negated + ? AnchorType.NOT_SENTENCE_BOUNDARY + : AnchorType.SENTENCE_BOUNDARY); + return true; + } restore(); return false; } diff --git a/third_party/joni/src/org/joni/SentenceBreakData.java b/third_party/joni/src/org/joni/SentenceBreakData.java new file mode 100644 index 0000000000..18ae82ae92 --- /dev/null +++ b/third_party/joni/src/org/joni/SentenceBreakData.java @@ -0,0 +1,681 @@ +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni; + +// Generated by dev/tools/generate_joni_boundary_data.pl. Do not edit manually. +final class SentenceBreakData { + static final String UNICODE_VERSION = "17.0.0"; + static final byte OTHER = 0; + static final byte ATERM = 1; + static final byte CLOSE = 2; + static final byte CR = 3; + static final byte EXTEND = 4; + static final byte FORMAT = 5; + static final byte LF = 6; + static final byte LOWER = 7; + static final byte NUMERIC = 8; + static final byte OLETTER = 9; + static final byte SCONTINUE = 10; + static final byte SEP = 11; + static final byte SP = 12; + static final byte STERM = 13; + static final byte UPPER = 14; + + private static final int[] STARTS = { + 0x0, 0x9, 0xA, 0xB, 0xD, 0xE, 0x20, 0x21, 0x22, 0x23, + 0x27, 0x28, 0x29, 0x2A, 0x2C, 0x2D, 0x2E, 0x2F, 0x30, 0x3A, + 0x3C, 0x3F, 0x40, 0x41, 0x5B, 0x5C, 0x5D, 0x5E, 0x61, 0x7B, + 0x7C, 0x7D, 0x7E, 0x85, 0x86, 0xA0, 0xA1, 0xAA, 0xAB, 0xAC, + 0xAD, 0xAE, 0xB5, 0xB6, 0xBA, 0xBB, 0xBC, 0xC0, 0xD7, 0xD8, + 0xDF, 0xF7, 0xF8, 0x100, 0x101, 0x102, 0x103, 0x104, 0x105, 0x106, + 0x107, 0x108, 0x109, 0x10A, 0x10B, 0x10C, 0x10D, 0x10E, 0x10F, 0x110, + 0x111, 0x112, 0x113, 0x114, 0x115, 0x116, 0x117, 0x118, 0x119, 0x11A, + 0x11B, 0x11C, 0x11D, 0x11E, 0x11F, 0x120, 0x121, 0x122, 0x123, 0x124, + 0x125, 0x126, 0x127, 0x128, 0x129, 0x12A, 0x12B, 0x12C, 0x12D, 0x12E, + 0x12F, 0x130, 0x131, 0x132, 0x133, 0x134, 0x135, 0x136, 0x137, 0x139, + 0x13A, 0x13B, 0x13C, 0x13D, 0x13E, 0x13F, 0x140, 0x141, 0x142, 0x143, + 0x144, 0x145, 0x146, 0x147, 0x148, 0x14A, 0x14B, 0x14C, 0x14D, 0x14E, + 0x14F, 0x150, 0x151, 0x152, 0x153, 0x154, 0x155, 0x156, 0x157, 0x158, + 0x159, 0x15A, 0x15B, 0x15C, 0x15D, 0x15E, 0x15F, 0x160, 0x161, 0x162, + 0x163, 0x164, 0x165, 0x166, 0x167, 0x168, 0x169, 0x16A, 0x16B, 0x16C, + 0x16D, 0x16E, 0x16F, 0x170, 0x171, 0x172, 0x173, 0x174, 0x175, 0x176, + 0x177, 0x178, 0x17A, 0x17B, 0x17C, 0x17D, 0x17E, 0x181, 0x183, 0x184, + 0x185, 0x186, 0x188, 0x189, 0x18C, 0x18E, 0x192, 0x193, 0x195, 0x196, + 0x199, 0x19C, 0x19E, 0x19F, 0x1A1, 0x1A2, 0x1A3, 0x1A4, 0x1A5, 0x1A6, + 0x1A8, 0x1A9, 0x1AA, 0x1AC, 0x1AD, 0x1AE, 0x1B0, 0x1B1, 0x1B4, 0x1B5, + 0x1B6, 0x1B7, 0x1B9, 0x1BB, 0x1BC, 0x1BD, 0x1C0, 0x1C4, 0x1C6, 0x1C7, + 0x1C9, 0x1CA, 0x1CC, 0x1CD, 0x1CE, 0x1CF, 0x1D0, 0x1D1, 0x1D2, 0x1D3, + 0x1D4, 0x1D5, 0x1D6, 0x1D7, 0x1D8, 0x1D9, 0x1DA, 0x1DB, 0x1DC, 0x1DE, + 0x1DF, 0x1E0, 0x1E1, 0x1E2, 0x1E3, 0x1E4, 0x1E5, 0x1E6, 0x1E7, 0x1E8, + 0x1E9, 0x1EA, 0x1EB, 0x1EC, 0x1ED, 0x1EE, 0x1EF, 0x1F1, 0x1F3, 0x1F4, + 0x1F5, 0x1F6, 0x1F9, 0x1FA, 0x1FB, 0x1FC, 0x1FD, 0x1FE, 0x1FF, 0x200, + 0x201, 0x202, 0x203, 0x204, 0x205, 0x206, 0x207, 0x208, 0x209, 0x20A, + 0x20B, 0x20C, 0x20D, 0x20E, 0x20F, 0x210, 0x211, 0x212, 0x213, 0x214, + 0x215, 0x216, 0x217, 0x218, 0x219, 0x21A, 0x21B, 0x21C, 0x21D, 0x21E, + 0x21F, 0x220, 0x221, 0x222, 0x223, 0x224, 0x225, 0x226, 0x227, 0x228, + 0x229, 0x22A, 0x22B, 0x22C, 0x22D, 0x22E, 0x22F, 0x230, 0x231, 0x232, + 0x233, 0x23A, 0x23C, 0x23D, 0x23F, 0x241, 0x242, 0x243, 0x247, 0x248, + 0x249, 0x24A, 0x24B, 0x24C, 0x24D, 0x24E, 0x24F, 0x294, 0x296, 0x2B0, + 0x2B9, 0x2C0, 0x2C2, 0x2C6, 0x2D2, 0x2E0, 0x2E5, 0x2EC, 0x2ED, 0x2EE, + 0x2EF, 0x300, 0x370, 0x371, 0x372, 0x373, 0x374, 0x375, 0x376, 0x377, + 0x378, 0x37A, 0x37B, 0x37E, 0x37F, 0x380, 0x386, 0x387, 0x388, 0x38B, + 0x38C, 0x38D, 0x38E, 0x390, 0x391, 0x3A2, 0x3A3, 0x3AC, 0x3CF, 0x3D0, + 0x3D2, 0x3D5, 0x3D8, 0x3D9, 0x3DA, 0x3DB, 0x3DC, 0x3DD, 0x3DE, 0x3DF, + 0x3E0, 0x3E1, 0x3E2, 0x3E3, 0x3E4, 0x3E5, 0x3E6, 0x3E7, 0x3E8, 0x3E9, + 0x3EA, 0x3EB, 0x3EC, 0x3ED, 0x3EE, 0x3EF, 0x3F4, 0x3F5, 0x3F6, 0x3F7, + 0x3F8, 0x3F9, 0x3FB, 0x3FD, 0x430, 0x460, 0x461, 0x462, 0x463, 0x464, + 0x465, 0x466, 0x467, 0x468, 0x469, 0x46A, 0x46B, 0x46C, 0x46D, 0x46E, + 0x46F, 0x470, 0x471, 0x472, 0x473, 0x474, 0x475, 0x476, 0x477, 0x478, + 0x479, 0x47A, 0x47B, 0x47C, 0x47D, 0x47E, 0x47F, 0x480, 0x481, 0x482, + 0x483, 0x488, 0x48A, 0x48B, 0x48C, 0x48D, 0x48E, 0x48F, 0x490, 0x491, + 0x492, 0x493, 0x494, 0x495, 0x496, 0x497, 0x498, 0x499, 0x49A, 0x49B, + 0x49C, 0x49D, 0x49E, 0x49F, 0x4A0, 0x4A1, 0x4A2, 0x4A3, 0x4A4, 0x4A5, + 0x4A6, 0x4A7, 0x4A8, 0x4A9, 0x4AA, 0x4AB, 0x4AC, 0x4AD, 0x4AE, 0x4AF, + 0x4B0, 0x4B1, 0x4B2, 0x4B3, 0x4B4, 0x4B5, 0x4B6, 0x4B7, 0x4B8, 0x4B9, + 0x4BA, 0x4BB, 0x4BC, 0x4BD, 0x4BE, 0x4BF, 0x4C0, 0x4C2, 0x4C3, 0x4C4, + 0x4C5, 0x4C6, 0x4C7, 0x4C8, 0x4C9, 0x4CA, 0x4CB, 0x4CC, 0x4CD, 0x4CE, + 0x4D0, 0x4D1, 0x4D2, 0x4D3, 0x4D4, 0x4D5, 0x4D6, 0x4D7, 0x4D8, 0x4D9, + 0x4DA, 0x4DB, 0x4DC, 0x4DD, 0x4DE, 0x4DF, 0x4E0, 0x4E1, 0x4E2, 0x4E3, + 0x4E4, 0x4E5, 0x4E6, 0x4E7, 0x4E8, 0x4E9, 0x4EA, 0x4EB, 0x4EC, 0x4ED, + 0x4EE, 0x4EF, 0x4F0, 0x4F1, 0x4F2, 0x4F3, 0x4F4, 0x4F5, 0x4F6, 0x4F7, + 0x4F8, 0x4F9, 0x4FA, 0x4FB, 0x4FC, 0x4FD, 0x4FE, 0x4FF, 0x500, 0x501, + 0x502, 0x503, 0x504, 0x505, 0x506, 0x507, 0x508, 0x509, 0x50A, 0x50B, + 0x50C, 0x50D, 0x50E, 0x50F, 0x510, 0x511, 0x512, 0x513, 0x514, 0x515, + 0x516, 0x517, 0x518, 0x519, 0x51A, 0x51B, 0x51C, 0x51D, 0x51E, 0x51F, + 0x520, 0x521, 0x522, 0x523, 0x524, 0x525, 0x526, 0x527, 0x528, 0x529, + 0x52A, 0x52B, 0x52C, 0x52D, 0x52E, 0x52F, 0x530, 0x531, 0x557, 0x559, + 0x55A, 0x55D, 0x55E, 0x560, 0x589, 0x58A, 0x591, 0x5BE, 0x5BF, 0x5C0, + 0x5C1, 0x5C3, 0x5C4, 0x5C6, 0x5C7, 0x5C8, 0x5D0, 0x5EB, 0x5EF, 0x5F3, + 0x5F4, 0x600, 0x606, 0x60C, 0x60E, 0x610, 0x61B, 0x61C, 0x61D, 0x620, + 0x640, 0x641, 0x64B, 0x660, 0x66A, 0x66B, 0x66D, 0x66E, 0x670, 0x671, + 0x6D4, 0x6D5, 0x6D6, 0x6DD, 0x6DE, 0x6DF, 0x6E5, 0x6E7, 0x6E9, 0x6EA, + 0x6EE, 0x6F0, 0x6FA, 0x6FD, 0x6FF, 0x700, 0x703, 0x70F, 0x710, 0x711, + 0x712, 0x730, 0x74B, 0x74D, 0x7A6, 0x7B1, 0x7B2, 0x7C0, 0x7CA, 0x7EB, + 0x7F4, 0x7F6, 0x7F8, 0x7F9, 0x7FA, 0x7FB, 0x7FD, 0x7FE, 0x800, 0x816, + 0x81A, 0x81B, 0x824, 0x825, 0x828, 0x829, 0x82E, 0x837, 0x838, 0x839, + 0x83A, 0x83D, 0x83F, 0x840, 0x859, 0x85C, 0x860, 0x86B, 0x870, 0x888, + 0x889, 0x890, 0x892, 0x897, 0x8A0, 0x8C9, 0x8CA, 0x8E2, 0x8E3, 0x903, + 0x904, 0x93A, 0x93B, 0x93C, 0x93D, 0x93E, 0x941, 0x949, 0x94D, 0x94E, + 0x950, 0x951, 0x958, 0x962, 0x964, 0x966, 0x970, 0x971, 0x972, 0x981, + 0x982, 0x984, 0x985, 0x98D, 0x98F, 0x991, 0x993, 0x9A9, 0x9AA, 0x9B1, + 0x9B2, 0x9B3, 0x9B6, 0x9BA, 0x9BC, 0x9BD, 0x9BE, 0x9C1, 0x9C5, 0x9C7, + 0x9C9, 0x9CB, 0x9CD, 0x9CE, 0x9CF, 0x9D7, 0x9D8, 0x9DC, 0x9DE, 0x9DF, + 0x9E2, 0x9E4, 0x9E6, 0x9F0, 0x9F2, 0x9FC, 0x9FD, 0x9FE, 0x9FF, 0xA01, + 0xA03, 0xA04, 0xA05, 0xA0B, 0xA0F, 0xA11, 0xA13, 0xA29, 0xA2A, 0xA31, + 0xA32, 0xA34, 0xA35, 0xA37, 0xA38, 0xA3A, 0xA3C, 0xA3D, 0xA3E, 0xA41, + 0xA43, 0xA47, 0xA49, 0xA4B, 0xA4E, 0xA51, 0xA52, 0xA59, 0xA5D, 0xA5E, + 0xA5F, 0xA66, 0xA70, 0xA72, 0xA75, 0xA76, 0xA81, 0xA83, 0xA84, 0xA85, + 0xA8E, 0xA8F, 0xA92, 0xA93, 0xAA9, 0xAAA, 0xAB1, 0xAB2, 0xAB4, 0xAB5, + 0xABA, 0xABC, 0xABD, 0xABE, 0xAC1, 0xAC6, 0xAC7, 0xAC9, 0xACA, 0xACB, + 0xACD, 0xACE, 0xAD0, 0xAD1, 0xAE0, 0xAE2, 0xAE4, 0xAE6, 0xAF0, 0xAF9, + 0xAFA, 0xB00, 0xB01, 0xB02, 0xB04, 0xB05, 0xB0D, 0xB0F, 0xB11, 0xB13, + 0xB29, 0xB2A, 0xB31, 0xB32, 0xB34, 0xB35, 0xB3A, 0xB3C, 0xB3D, 0xB3E, + 0xB3F, 0xB40, 0xB41, 0xB45, 0xB47, 0xB49, 0xB4B, 0xB4D, 0xB4E, 0xB55, + 0xB57, 0xB58, 0xB5C, 0xB5E, 0xB5F, 0xB62, 0xB64, 0xB66, 0xB70, 0xB71, + 0xB72, 0xB82, 0xB83, 0xB84, 0xB85, 0xB8B, 0xB8E, 0xB91, 0xB92, 0xB96, + 0xB99, 0xB9B, 0xB9C, 0xB9D, 0xB9E, 0xBA0, 0xBA3, 0xBA5, 0xBA8, 0xBAB, + 0xBAE, 0xBBA, 0xBBE, 0xBC0, 0xBC1, 0xBC3, 0xBC6, 0xBC9, 0xBCA, 0xBCD, + 0xBCE, 0xBD0, 0xBD1, 0xBD7, 0xBD8, 0xBE6, 0xBF0, 0xC00, 0xC01, 0xC04, + 0xC05, 0xC0D, 0xC0E, 0xC11, 0xC12, 0xC29, 0xC2A, 0xC3A, 0xC3C, 0xC3D, + 0xC3E, 0xC41, 0xC45, 0xC46, 0xC49, 0xC4A, 0xC4E, 0xC55, 0xC57, 0xC58, + 0xC5B, 0xC5C, 0xC5E, 0xC60, 0xC62, 0xC64, 0xC66, 0xC70, 0xC80, 0xC81, + 0xC82, 0xC84, 0xC85, 0xC8D, 0xC8E, 0xC91, 0xC92, 0xCA9, 0xCAA, 0xCB4, + 0xCB5, 0xCBA, 0xCBC, 0xCBD, 0xCBE, 0xCBF, 0xCC0, 0xCC5, 0xCC6, 0xCC7, + 0xCC9, 0xCCA, 0xCCC, 0xCCE, 0xCD5, 0xCD7, 0xCDC, 0xCDF, 0xCE0, 0xCE2, + 0xCE4, 0xCE6, 0xCF0, 0xCF1, 0xCF3, 0xCF4, 0xD00, 0xD02, 0xD04, 0xD0D, + 0xD0E, 0xD11, 0xD12, 0xD3B, 0xD3D, 0xD3E, 0xD41, 0xD45, 0xD46, 0xD49, + 0xD4A, 0xD4D, 0xD4E, 0xD4F, 0xD54, 0xD57, 0xD58, 0xD5F, 0xD62, 0xD64, + 0xD66, 0xD70, 0xD7A, 0xD80, 0xD81, 0xD82, 0xD84, 0xD85, 0xD97, 0xD9A, + 0xDB2, 0xDB3, 0xDBC, 0xDBD, 0xDBE, 0xDC0, 0xDC7, 0xDCA, 0xDCB, 0xDCF, + 0xDD2, 0xDD5, 0xDD6, 0xDD7, 0xDD8, 0xDE0, 0xDE6, 0xDF0, 0xDF2, 0xDF4, + 0xE01, 0xE31, 0xE32, 0xE34, 0xE3B, 0xE40, 0xE46, 0xE47, 0xE4F, 0xE50, + 0xE5A, 0xE81, 0xE83, 0xE84, 0xE85, 0xE86, 0xE8B, 0xE8C, 0xEA4, 0xEA5, + 0xEA6, 0xEA7, 0xEB1, 0xEB2, 0xEB4, 0xEBD, 0xEBE, 0xEC0, 0xEC5, 0xEC6, + 0xEC7, 0xEC8, 0xECF, 0xED0, 0xEDA, 0xEDC, 0xEE0, 0xF00, 0xF01, 0xF18, + 0xF1A, 0xF20, 0xF2A, 0xF35, 0xF36, 0xF37, 0xF38, 0xF39, 0xF3A, 0xF3B, + 0xF3C, 0xF3D, 0xF3E, 0xF40, 0xF48, 0xF49, 0xF6D, 0xF71, 0xF7F, 0xF80, + 0xF85, 0xF86, 0xF88, 0xF8D, 0xF98, 0xF99, 0xFBD, 0xFC6, 0xFC7, 0x1000, + 0x102B, 0x102D, 0x1031, 0x1032, 0x1038, 0x1039, 0x103B, 0x103D, 0x103F, 0x1040, + 0x104A, 0x104C, 0x1050, 0x1056, 0x1058, 0x105A, 0x105E, 0x1061, 0x1062, 0x1065, + 0x1067, 0x106E, 0x1071, 0x1075, 0x1082, 0x1083, 0x1085, 0x1087, 0x108D, 0x108E, + 0x108F, 0x1090, 0x109A, 0x109D, 0x109E, 0x10A0, 0x10C6, 0x10C7, 0x10C8, 0x10CD, + 0x10CE, 0x10D0, 0x10FB, 0x10FC, 0x10FD, 0x1100, 0x1249, 0x124A, 0x124E, 0x1250, + 0x1257, 0x1258, 0x1259, 0x125A, 0x125E, 0x1260, 0x1289, 0x128A, 0x128E, 0x1290, + 0x12B1, 0x12B2, 0x12B6, 0x12B8, 0x12BF, 0x12C0, 0x12C1, 0x12C2, 0x12C6, 0x12C8, + 0x12D7, 0x12D8, 0x1311, 0x1312, 0x1316, 0x1318, 0x135B, 0x135D, 0x1360, 0x1362, + 0x1363, 0x1367, 0x1369, 0x1380, 0x1390, 0x13A0, 0x13F6, 0x13F8, 0x13FE, 0x1401, + 0x166D, 0x166E, 0x166F, 0x1680, 0x1681, 0x169B, 0x169C, 0x169D, 0x16A0, 0x16EB, + 0x16EE, 0x16F1, 0x16F9, 0x1700, 0x1712, 0x1715, 0x1716, 0x171F, 0x1732, 0x1734, + 0x1735, 0x1737, 0x1740, 0x1752, 0x1754, 0x1760, 0x176D, 0x176E, 0x1771, 0x1772, + 0x1774, 0x1780, 0x17B4, 0x17B6, 0x17B7, 0x17BE, 0x17C6, 0x17C7, 0x17C9, 0x17D4, + 0x17D6, 0x17D7, 0x17D8, 0x17DC, 0x17DD, 0x17DE, 0x17E0, 0x17EA, 0x1802, 0x1803, + 0x1804, 0x1808, 0x1809, 0x180A, 0x180B, 0x180E, 0x180F, 0x1810, 0x181A, 0x1820, + 0x1843, 0x1844, 0x1879, 0x1880, 0x1885, 0x1887, 0x18A9, 0x18AA, 0x18AB, 0x18B0, + 0x18F6, 0x1900, 0x191F, 0x1920, 0x1923, 0x1927, 0x1929, 0x192C, 0x1930, 0x1932, + 0x1933, 0x1939, 0x193C, 0x1944, 0x1946, 0x1950, 0x196E, 0x1970, 0x1975, 0x1980, + 0x19AC, 0x19B0, 0x19CA, 0x19D0, 0x19DA, 0x19DB, 0x1A00, 0x1A17, 0x1A19, 0x1A1B, + 0x1A1C, 0x1A20, 0x1A55, 0x1A56, 0x1A57, 0x1A58, 0x1A5F, 0x1A60, 0x1A61, 0x1A62, + 0x1A63, 0x1A65, 0x1A6D, 0x1A73, 0x1A7D, 0x1A7F, 0x1A80, 0x1A8A, 0x1A90, 0x1A9A, + 0x1AA7, 0x1AA8, 0x1AAC, 0x1AB0, 0x1ABE, 0x1ABF, 0x1ADE, 0x1AE0, 0x1AEC, 0x1B00, + 0x1B04, 0x1B05, 0x1B34, 0x1B35, 0x1B36, 0x1B3B, 0x1B3C, 0x1B3D, 0x1B42, 0x1B43, + 0x1B45, 0x1B4D, 0x1B4E, 0x1B50, 0x1B5A, 0x1B5C, 0x1B5E, 0x1B60, 0x1B6B, 0x1B74, + 0x1B7D, 0x1B80, 0x1B82, 0x1B83, 0x1BA1, 0x1BA2, 0x1BA6, 0x1BA8, 0x1BAA, 0x1BAB, + 0x1BAE, 0x1BB0, 0x1BBA, 0x1BE6, 0x1BE7, 0x1BE8, 0x1BEA, 0x1BED, 0x1BEE, 0x1BEF, + 0x1BF2, 0x1BF4, 0x1C00, 0x1C24, 0x1C2C, 0x1C34, 0x1C36, 0x1C38, 0x1C3B, 0x1C3D, + 0x1C40, 0x1C4A, 0x1C4D, 0x1C50, 0x1C5A, 0x1C78, 0x1C7E, 0x1C80, 0x1C89, 0x1C8A, + 0x1C8B, 0x1C90, 0x1CBB, 0x1CBD, 0x1CC0, 0x1CD0, 0x1CD3, 0x1CD4, 0x1CE1, 0x1CE2, + 0x1CE9, 0x1CED, 0x1CEE, 0x1CF4, 0x1CF5, 0x1CF7, 0x1CF8, 0x1CFA, 0x1CFB, 0x1D00, + 0x1D2C, 0x1D6B, 0x1D78, 0x1D79, 0x1D9B, 0x1DC0, 0x1E00, 0x1E01, 0x1E02, 0x1E03, + 0x1E04, 0x1E05, 0x1E06, 0x1E07, 0x1E08, 0x1E09, 0x1E0A, 0x1E0B, 0x1E0C, 0x1E0D, + 0x1E0E, 0x1E0F, 0x1E10, 0x1E11, 0x1E12, 0x1E13, 0x1E14, 0x1E15, 0x1E16, 0x1E17, + 0x1E18, 0x1E19, 0x1E1A, 0x1E1B, 0x1E1C, 0x1E1D, 0x1E1E, 0x1E1F, 0x1E20, 0x1E21, + 0x1E22, 0x1E23, 0x1E24, 0x1E25, 0x1E26, 0x1E27, 0x1E28, 0x1E29, 0x1E2A, 0x1E2B, + 0x1E2C, 0x1E2D, 0x1E2E, 0x1E2F, 0x1E30, 0x1E31, 0x1E32, 0x1E33, 0x1E34, 0x1E35, + 0x1E36, 0x1E37, 0x1E38, 0x1E39, 0x1E3A, 0x1E3B, 0x1E3C, 0x1E3D, 0x1E3E, 0x1E3F, + 0x1E40, 0x1E41, 0x1E42, 0x1E43, 0x1E44, 0x1E45, 0x1E46, 0x1E47, 0x1E48, 0x1E49, + 0x1E4A, 0x1E4B, 0x1E4C, 0x1E4D, 0x1E4E, 0x1E4F, 0x1E50, 0x1E51, 0x1E52, 0x1E53, + 0x1E54, 0x1E55, 0x1E56, 0x1E57, 0x1E58, 0x1E59, 0x1E5A, 0x1E5B, 0x1E5C, 0x1E5D, + 0x1E5E, 0x1E5F, 0x1E60, 0x1E61, 0x1E62, 0x1E63, 0x1E64, 0x1E65, 0x1E66, 0x1E67, + 0x1E68, 0x1E69, 0x1E6A, 0x1E6B, 0x1E6C, 0x1E6D, 0x1E6E, 0x1E6F, 0x1E70, 0x1E71, + 0x1E72, 0x1E73, 0x1E74, 0x1E75, 0x1E76, 0x1E77, 0x1E78, 0x1E79, 0x1E7A, 0x1E7B, + 0x1E7C, 0x1E7D, 0x1E7E, 0x1E7F, 0x1E80, 0x1E81, 0x1E82, 0x1E83, 0x1E84, 0x1E85, + 0x1E86, 0x1E87, 0x1E88, 0x1E89, 0x1E8A, 0x1E8B, 0x1E8C, 0x1E8D, 0x1E8E, 0x1E8F, + 0x1E90, 0x1E91, 0x1E92, 0x1E93, 0x1E94, 0x1E95, 0x1E9E, 0x1E9F, 0x1EA0, 0x1EA1, + 0x1EA2, 0x1EA3, 0x1EA4, 0x1EA5, 0x1EA6, 0x1EA7, 0x1EA8, 0x1EA9, 0x1EAA, 0x1EAB, + 0x1EAC, 0x1EAD, 0x1EAE, 0x1EAF, 0x1EB0, 0x1EB1, 0x1EB2, 0x1EB3, 0x1EB4, 0x1EB5, + 0x1EB6, 0x1EB7, 0x1EB8, 0x1EB9, 0x1EBA, 0x1EBB, 0x1EBC, 0x1EBD, 0x1EBE, 0x1EBF, + 0x1EC0, 0x1EC1, 0x1EC2, 0x1EC3, 0x1EC4, 0x1EC5, 0x1EC6, 0x1EC7, 0x1EC8, 0x1EC9, + 0x1ECA, 0x1ECB, 0x1ECC, 0x1ECD, 0x1ECE, 0x1ECF, 0x1ED0, 0x1ED1, 0x1ED2, 0x1ED3, + 0x1ED4, 0x1ED5, 0x1ED6, 0x1ED7, 0x1ED8, 0x1ED9, 0x1EDA, 0x1EDB, 0x1EDC, 0x1EDD, + 0x1EDE, 0x1EDF, 0x1EE0, 0x1EE1, 0x1EE2, 0x1EE3, 0x1EE4, 0x1EE5, 0x1EE6, 0x1EE7, + 0x1EE8, 0x1EE9, 0x1EEA, 0x1EEB, 0x1EEC, 0x1EED, 0x1EEE, 0x1EEF, 0x1EF0, 0x1EF1, + 0x1EF2, 0x1EF3, 0x1EF4, 0x1EF5, 0x1EF6, 0x1EF7, 0x1EF8, 0x1EF9, 0x1EFA, 0x1EFB, + 0x1EFC, 0x1EFD, 0x1EFE, 0x1EFF, 0x1F08, 0x1F10, 0x1F16, 0x1F18, 0x1F1E, 0x1F20, + 0x1F28, 0x1F30, 0x1F38, 0x1F40, 0x1F46, 0x1F48, 0x1F4E, 0x1F50, 0x1F58, 0x1F59, + 0x1F5A, 0x1F5B, 0x1F5C, 0x1F5D, 0x1F5E, 0x1F5F, 0x1F60, 0x1F68, 0x1F70, 0x1F7E, + 0x1F80, 0x1F88, 0x1F90, 0x1F98, 0x1FA0, 0x1FA8, 0x1FB0, 0x1FB5, 0x1FB6, 0x1FB8, + 0x1FBD, 0x1FBE, 0x1FBF, 0x1FC2, 0x1FC5, 0x1FC6, 0x1FC8, 0x1FCD, 0x1FD0, 0x1FD4, + 0x1FD6, 0x1FD8, 0x1FDC, 0x1FE0, 0x1FE8, 0x1FED, 0x1FF2, 0x1FF5, 0x1FF6, 0x1FF8, + 0x1FFD, 0x2000, 0x200B, 0x200C, 0x200E, 0x2010, 0x2013, 0x2015, 0x2018, 0x2019, + 0x201A, 0x201B, 0x201D, 0x201E, 0x201F, 0x2020, 0x2024, 0x2025, 0x2028, 0x2029, + 0x202A, 0x202F, 0x2030, 0x2039, 0x203A, 0x203B, 0x203C, 0x203E, 0x2045, 0x2046, + 0x2047, 0x204A, 0x205F, 0x2060, 0x2065, 0x2066, 0x2070, 0x2071, 0x2072, 0x207D, + 0x207E, 0x207F, 0x2080, 0x208D, 0x208E, 0x208F, 0x2090, 0x209D, 0x20D0, 0x20DD, + 0x20E1, 0x20E2, 0x20E5, 0x20F1, 0x2102, 0x2103, 0x2107, 0x2108, 0x210A, 0x210B, + 0x210E, 0x2110, 0x2113, 0x2114, 0x2115, 0x2116, 0x2119, 0x211E, 0x2124, 0x2125, + 0x2126, 0x2127, 0x2128, 0x2129, 0x212A, 0x212E, 0x212F, 0x2130, 0x2134, 0x2135, + 0x2139, 0x213A, 0x213C, 0x213E, 0x2140, 0x2145, 0x2146, 0x214A, 0x214E, 0x214F, + 0x2160, 0x2170, 0x2180, 0x2183, 0x2184, 0x2185, 0x2189, 0x2308, 0x2309, 0x230A, + 0x230B, 0x230C, 0x2329, 0x232A, 0x232B, 0x24B6, 0x24D0, 0x24EA, 0x275B, 0x2761, + 0x2768, 0x2769, 0x276A, 0x276B, 0x276C, 0x276D, 0x276E, 0x276F, 0x2770, 0x2771, + 0x2772, 0x2773, 0x2774, 0x2775, 0x2776, 0x27C5, 0x27C6, 0x27C7, 0x27E6, 0x27E7, + 0x27E8, 0x27E9, 0x27EA, 0x27EB, 0x27EC, 0x27ED, 0x27EE, 0x27EF, 0x27F0, 0x2983, + 0x2984, 0x2985, 0x2986, 0x2987, 0x2988, 0x2989, 0x298A, 0x298B, 0x298C, 0x298D, + 0x298E, 0x298F, 0x2990, 0x2991, 0x2992, 0x2993, 0x2994, 0x2995, 0x2996, 0x2997, + 0x2998, 0x2999, 0x29D8, 0x29D9, 0x29DA, 0x29DB, 0x29DC, 0x29FC, 0x29FD, 0x29FE, + 0x2C00, 0x2C30, 0x2C60, 0x2C61, 0x2C62, 0x2C65, 0x2C67, 0x2C68, 0x2C69, 0x2C6A, + 0x2C6B, 0x2C6C, 0x2C6D, 0x2C71, 0x2C72, 0x2C73, 0x2C75, 0x2C76, 0x2C7C, 0x2C7E, + 0x2C81, 0x2C82, 0x2C83, 0x2C84, 0x2C85, 0x2C86, 0x2C87, 0x2C88, 0x2C89, 0x2C8A, + 0x2C8B, 0x2C8C, 0x2C8D, 0x2C8E, 0x2C8F, 0x2C90, 0x2C91, 0x2C92, 0x2C93, 0x2C94, + 0x2C95, 0x2C96, 0x2C97, 0x2C98, 0x2C99, 0x2C9A, 0x2C9B, 0x2C9C, 0x2C9D, 0x2C9E, + 0x2C9F, 0x2CA0, 0x2CA1, 0x2CA2, 0x2CA3, 0x2CA4, 0x2CA5, 0x2CA6, 0x2CA7, 0x2CA8, + 0x2CA9, 0x2CAA, 0x2CAB, 0x2CAC, 0x2CAD, 0x2CAE, 0x2CAF, 0x2CB0, 0x2CB1, 0x2CB2, + 0x2CB3, 0x2CB4, 0x2CB5, 0x2CB6, 0x2CB7, 0x2CB8, 0x2CB9, 0x2CBA, 0x2CBB, 0x2CBC, + 0x2CBD, 0x2CBE, 0x2CBF, 0x2CC0, 0x2CC1, 0x2CC2, 0x2CC3, 0x2CC4, 0x2CC5, 0x2CC6, + 0x2CC7, 0x2CC8, 0x2CC9, 0x2CCA, 0x2CCB, 0x2CCC, 0x2CCD, 0x2CCE, 0x2CCF, 0x2CD0, + 0x2CD1, 0x2CD2, 0x2CD3, 0x2CD4, 0x2CD5, 0x2CD6, 0x2CD7, 0x2CD8, 0x2CD9, 0x2CDA, + 0x2CDB, 0x2CDC, 0x2CDD, 0x2CDE, 0x2CDF, 0x2CE0, 0x2CE1, 0x2CE2, 0x2CE3, 0x2CE5, + 0x2CEB, 0x2CEC, 0x2CED, 0x2CEE, 0x2CEF, 0x2CF2, 0x2CF3, 0x2CF4, 0x2CF9, 0x2CFC, + 0x2D00, 0x2D26, 0x2D27, 0x2D28, 0x2D2D, 0x2D2E, 0x2D30, 0x2D68, 0x2D6F, 0x2D70, + 0x2D7F, 0x2D80, 0x2D97, 0x2DA0, 0x2DA7, 0x2DA8, 0x2DAF, 0x2DB0, 0x2DB7, 0x2DB8, + 0x2DBF, 0x2DC0, 0x2DC7, 0x2DC8, 0x2DCF, 0x2DD0, 0x2DD7, 0x2DD8, 0x2DDF, 0x2DE0, + 0x2E00, 0x2E02, 0x2E03, 0x2E04, 0x2E05, 0x2E06, 0x2E09, 0x2E0A, 0x2E0B, 0x2E0C, + 0x2E0D, 0x2E0E, 0x2E1C, 0x2E1D, 0x2E1E, 0x2E20, 0x2E21, 0x2E22, 0x2E23, 0x2E24, + 0x2E25, 0x2E26, 0x2E27, 0x2E28, 0x2E29, 0x2E2A, 0x2E2E, 0x2E2F, 0x2E30, 0x2E3C, + 0x2E3D, 0x2E42, 0x2E43, 0x2E53, 0x2E55, 0x2E56, 0x2E57, 0x2E58, 0x2E59, 0x2E5A, + 0x2E5B, 0x2E5C, 0x2E5D, 0x3000, 0x3001, 0x3002, 0x3003, 0x3005, 0x3006, 0x3007, + 0x3008, 0x3009, 0x300A, 0x300B, 0x300C, 0x300D, 0x300E, 0x300F, 0x3010, 0x3011, + 0x3012, 0x3014, 0x3015, 0x3016, 0x3017, 0x3018, 0x3019, 0x301A, 0x301B, 0x301C, + 0x301D, 0x301E, 0x3020, 0x3021, 0x302A, 0x302E, 0x3030, 0x3031, 0x3036, 0x3038, + 0x303B, 0x303C, 0x303D, 0x3041, 0x3097, 0x3099, 0x309B, 0x309D, 0x309F, 0x30A0, + 0x30A1, 0x30FB, 0x30FC, 0x30FF, 0x3100, 0x3105, 0x3130, 0x3131, 0x318F, 0x31A0, + 0x31C0, 0x31F0, 0x3200, 0x3400, 0x4DC0, 0x4E00, 0xA015, 0xA016, 0xA48D, 0xA4D0, + 0xA4F8, 0xA4FE, 0xA4FF, 0xA500, 0xA60C, 0xA60D, 0xA60E, 0xA610, 0xA620, 0xA62A, + 0xA62C, 0xA640, 0xA641, 0xA642, 0xA643, 0xA644, 0xA645, 0xA646, 0xA647, 0xA648, + 0xA649, 0xA64A, 0xA64B, 0xA64C, 0xA64D, 0xA64E, 0xA64F, 0xA650, 0xA651, 0xA652, + 0xA653, 0xA654, 0xA655, 0xA656, 0xA657, 0xA658, 0xA659, 0xA65A, 0xA65B, 0xA65C, + 0xA65D, 0xA65E, 0xA65F, 0xA660, 0xA661, 0xA662, 0xA663, 0xA664, 0xA665, 0xA666, + 0xA667, 0xA668, 0xA669, 0xA66A, 0xA66B, 0xA66C, 0xA66D, 0xA66E, 0xA66F, 0xA670, + 0xA673, 0xA674, 0xA67E, 0xA67F, 0xA680, 0xA681, 0xA682, 0xA683, 0xA684, 0xA685, + 0xA686, 0xA687, 0xA688, 0xA689, 0xA68A, 0xA68B, 0xA68C, 0xA68D, 0xA68E, 0xA68F, + 0xA690, 0xA691, 0xA692, 0xA693, 0xA694, 0xA695, 0xA696, 0xA697, 0xA698, 0xA699, + 0xA69A, 0xA69B, 0xA69C, 0xA69E, 0xA6A0, 0xA6E6, 0xA6F0, 0xA6F2, 0xA6F3, 0xA6F4, + 0xA6F7, 0xA6F8, 0xA717, 0xA720, 0xA722, 0xA723, 0xA724, 0xA725, 0xA726, 0xA727, + 0xA728, 0xA729, 0xA72A, 0xA72B, 0xA72C, 0xA72D, 0xA72E, 0xA72F, 0xA732, 0xA733, + 0xA734, 0xA735, 0xA736, 0xA737, 0xA738, 0xA739, 0xA73A, 0xA73B, 0xA73C, 0xA73D, + 0xA73E, 0xA73F, 0xA740, 0xA741, 0xA742, 0xA743, 0xA744, 0xA745, 0xA746, 0xA747, + 0xA748, 0xA749, 0xA74A, 0xA74B, 0xA74C, 0xA74D, 0xA74E, 0xA74F, 0xA750, 0xA751, + 0xA752, 0xA753, 0xA754, 0xA755, 0xA756, 0xA757, 0xA758, 0xA759, 0xA75A, 0xA75B, + 0xA75C, 0xA75D, 0xA75E, 0xA75F, 0xA760, 0xA761, 0xA762, 0xA763, 0xA764, 0xA765, + 0xA766, 0xA767, 0xA768, 0xA769, 0xA76A, 0xA76B, 0xA76C, 0xA76D, 0xA76E, 0xA76F, + 0xA770, 0xA771, 0xA779, 0xA77A, 0xA77B, 0xA77C, 0xA77D, 0xA77F, 0xA780, 0xA781, + 0xA782, 0xA783, 0xA784, 0xA785, 0xA786, 0xA787, 0xA788, 0xA789, 0xA78B, 0xA78C, + 0xA78D, 0xA78E, 0xA78F, 0xA790, 0xA791, 0xA792, 0xA793, 0xA796, 0xA797, 0xA798, + 0xA799, 0xA79A, 0xA79B, 0xA79C, 0xA79D, 0xA79E, 0xA79F, 0xA7A0, 0xA7A1, 0xA7A2, + 0xA7A3, 0xA7A4, 0xA7A5, 0xA7A6, 0xA7A7, 0xA7A8, 0xA7A9, 0xA7AA, 0xA7AF, 0xA7B0, + 0xA7B5, 0xA7B6, 0xA7B7, 0xA7B8, 0xA7B9, 0xA7BA, 0xA7BB, 0xA7BC, 0xA7BD, 0xA7BE, + 0xA7BF, 0xA7C0, 0xA7C1, 0xA7C2, 0xA7C3, 0xA7C4, 0xA7C8, 0xA7C9, 0xA7CA, 0xA7CB, + 0xA7CD, 0xA7CE, 0xA7CF, 0xA7D0, 0xA7D1, 0xA7D2, 0xA7D3, 0xA7D4, 0xA7D5, 0xA7D6, + 0xA7D7, 0xA7D8, 0xA7D9, 0xA7DA, 0xA7DB, 0xA7DC, 0xA7DD, 0xA7F1, 0xA7F5, 0xA7F6, + 0xA7F7, 0xA7F8, 0xA7FA, 0xA7FB, 0xA802, 0xA803, 0xA806, 0xA807, 0xA80B, 0xA80C, + 0xA823, 0xA825, 0xA827, 0xA828, 0xA82C, 0xA82D, 0xA840, 0xA874, 0xA876, 0xA878, + 0xA880, 0xA882, 0xA8B4, 0xA8C4, 0xA8C6, 0xA8CE, 0xA8D0, 0xA8DA, 0xA8E0, 0xA8F2, + 0xA8F8, 0xA8FB, 0xA8FC, 0xA8FD, 0xA8FF, 0xA900, 0xA90A, 0xA926, 0xA92E, 0xA92F, + 0xA930, 0xA947, 0xA952, 0xA954, 0xA960, 0xA97D, 0xA980, 0xA983, 0xA984, 0xA9B3, + 0xA9B4, 0xA9B6, 0xA9BA, 0xA9BC, 0xA9BE, 0xA9C1, 0xA9C8, 0xA9CA, 0xA9CF, 0xA9D0, + 0xA9DA, 0xA9E0, 0xA9E5, 0xA9E6, 0xA9E7, 0xA9F0, 0xA9FA, 0xA9FF, 0xAA00, 0xAA29, + 0xAA2F, 0xAA31, 0xAA33, 0xAA35, 0xAA37, 0xAA40, 0xAA43, 0xAA44, 0xAA4C, 0xAA4D, + 0xAA4E, 0xAA50, 0xAA5A, 0xAA5D, 0xAA60, 0xAA70, 0xAA71, 0xAA77, 0xAA7A, 0xAA7B, + 0xAA7C, 0xAA7D, 0xAA7E, 0xAAB0, 0xAAB1, 0xAAB2, 0xAAB5, 0xAAB7, 0xAAB9, 0xAABE, + 0xAAC0, 0xAAC1, 0xAAC2, 0xAAC3, 0xAADB, 0xAADD, 0xAADE, 0xAAE0, 0xAAEB, 0xAAEC, + 0xAAEE, 0xAAF0, 0xAAF2, 0xAAF3, 0xAAF5, 0xAAF6, 0xAAF7, 0xAB01, 0xAB07, 0xAB09, + 0xAB0F, 0xAB11, 0xAB17, 0xAB20, 0xAB27, 0xAB28, 0xAB2F, 0xAB30, 0xAB5B, 0xAB5C, + 0xAB60, 0xAB69, 0xAB6A, 0xAB70, 0xABC0, 0xABE3, 0xABE5, 0xABE6, 0xABE8, 0xABE9, + 0xABEB, 0xABEC, 0xABED, 0xABEE, 0xABF0, 0xABFA, 0xAC00, 0xD7A4, 0xD7B0, 0xD7C7, + 0xD7CB, 0xD7FC, 0xF900, 0xFA6E, 0xFA70, 0xFADA, 0xFB00, 0xFB07, 0xFB13, 0xFB18, + 0xFB1D, 0xFB1E, 0xFB1F, 0xFB29, 0xFB2A, 0xFB37, 0xFB38, 0xFB3D, 0xFB3E, 0xFB3F, + 0xFB40, 0xFB42, 0xFB43, 0xFB45, 0xFB46, 0xFBB2, 0xFBD3, 0xFD3E, 0xFD3F, 0xFD40, + 0xFD50, 0xFD90, 0xFD92, 0xFDC8, 0xFDF0, 0xFDFC, 0xFE00, 0xFE10, 0xFE12, 0xFE13, + 0xFE15, 0xFE17, 0xFE18, 0xFE19, 0xFE20, 0xFE30, 0xFE31, 0xFE33, 0xFE35, 0xFE36, + 0xFE37, 0xFE38, 0xFE39, 0xFE3A, 0xFE3B, 0xFE3C, 0xFE3D, 0xFE3E, 0xFE3F, 0xFE40, + 0xFE41, 0xFE42, 0xFE43, 0xFE44, 0xFE45, 0xFE47, 0xFE48, 0xFE49, 0xFE50, 0xFE52, + 0xFE53, 0xFE54, 0xFE56, 0xFE58, 0xFE59, 0xFE5A, 0xFE5B, 0xFE5C, 0xFE5D, 0xFE5E, + 0xFE5F, 0xFE63, 0xFE64, 0xFE70, 0xFE75, 0xFE76, 0xFEFD, 0xFEFF, 0xFF00, 0xFF01, + 0xFF02, 0xFF08, 0xFF09, 0xFF0A, 0xFF0C, 0xFF0D, 0xFF0E, 0xFF0F, 0xFF10, 0xFF1A, + 0xFF1C, 0xFF1F, 0xFF20, 0xFF21, 0xFF3B, 0xFF3C, 0xFF3D, 0xFF3E, 0xFF41, 0xFF5B, + 0xFF5C, 0xFF5D, 0xFF5E, 0xFF5F, 0xFF60, 0xFF61, 0xFF62, 0xFF63, 0xFF64, 0xFF65, + 0xFF66, 0xFF70, 0xFF71, 0xFF9E, 0xFFA0, 0xFFBF, 0xFFC2, 0xFFC8, 0xFFCA, 0xFFD0, + 0xFFD2, 0xFFD8, 0xFFDA, 0xFFDD, 0xFFF9, 0xFFFC, 0x10000, 0x1000C, 0x1000D, 0x10027, + 0x10028, 0x1003B, 0x1003C, 0x1003E, 0x1003F, 0x1004E, 0x10050, 0x1005E, 0x10080, 0x100FB, + 0x10140, 0x10175, 0x101FD, 0x101FE, 0x10280, 0x1029D, 0x102A0, 0x102D1, 0x102E0, 0x102E1, + 0x10300, 0x10320, 0x1032D, 0x10341, 0x10342, 0x1034A, 0x1034B, 0x10350, 0x10376, 0x1037B, + 0x10380, 0x1039E, 0x103A0, 0x103C4, 0x103C8, 0x103D0, 0x103D1, 0x103D6, 0x10400, 0x10428, + 0x10450, 0x1049E, 0x104A0, 0x104AA, 0x104B0, 0x104D4, 0x104D8, 0x104FC, 0x10500, 0x10528, + 0x10530, 0x10564, 0x10570, 0x1057B, 0x1057C, 0x1058B, 0x1058C, 0x10593, 0x10594, 0x10596, + 0x10597, 0x105A2, 0x105A3, 0x105B2, 0x105B3, 0x105BA, 0x105BB, 0x105BD, 0x105C0, 0x105F4, + 0x10600, 0x10737, 0x10740, 0x10756, 0x10760, 0x10768, 0x10780, 0x10781, 0x10783, 0x10786, + 0x10787, 0x107B1, 0x107B2, 0x107BB, 0x10800, 0x10806, 0x10808, 0x10809, 0x1080A, 0x10836, + 0x10837, 0x10839, 0x1083C, 0x1083D, 0x1083F, 0x10856, 0x10860, 0x10877, 0x10880, 0x1089F, + 0x108E0, 0x108F3, 0x108F4, 0x108F6, 0x10900, 0x10916, 0x10920, 0x1093A, 0x10940, 0x1095A, + 0x10980, 0x109B8, 0x109BE, 0x109C0, 0x10A00, 0x10A01, 0x10A04, 0x10A05, 0x10A07, 0x10A0C, + 0x10A10, 0x10A14, 0x10A15, 0x10A18, 0x10A19, 0x10A36, 0x10A38, 0x10A3B, 0x10A3F, 0x10A40, + 0x10A56, 0x10A58, 0x10A60, 0x10A7D, 0x10A80, 0x10A9D, 0x10AC0, 0x10AC8, 0x10AC9, 0x10AE5, + 0x10AE7, 0x10B00, 0x10B36, 0x10B40, 0x10B56, 0x10B60, 0x10B73, 0x10B80, 0x10B92, 0x10C00, + 0x10C49, 0x10C80, 0x10CB3, 0x10CC0, 0x10CF3, 0x10D00, 0x10D24, 0x10D28, 0x10D30, 0x10D3A, + 0x10D40, 0x10D4A, 0x10D4E, 0x10D4F, 0x10D50, 0x10D66, 0x10D69, 0x10D6E, 0x10D6F, 0x10D70, + 0x10D86, 0x10E80, 0x10EAA, 0x10EAB, 0x10EAD, 0x10EB0, 0x10EB2, 0x10EC2, 0x10EC5, 0x10EC6, + 0x10EC8, 0x10EFA, 0x10F00, 0x10F1D, 0x10F27, 0x10F28, 0x10F30, 0x10F46, 0x10F51, 0x10F55, + 0x10F5A, 0x10F70, 0x10F82, 0x10F86, 0x10F8A, 0x10FB0, 0x10FC5, 0x10FE0, 0x10FF7, 0x11000, + 0x11001, 0x11002, 0x11003, 0x11038, 0x11047, 0x11049, 0x11066, 0x11070, 0x11071, 0x11073, + 0x11075, 0x11076, 0x1107F, 0x11082, 0x11083, 0x110B0, 0x110B3, 0x110B7, 0x110B9, 0x110BB, + 0x110BD, 0x110BE, 0x110C2, 0x110C3, 0x110CD, 0x110CE, 0x110D0, 0x110E9, 0x110F0, 0x110FA, + 0x11100, 0x11103, 0x11127, 0x1112C, 0x1112D, 0x11135, 0x11136, 0x11140, 0x11141, 0x11144, + 0x11145, 0x11147, 0x11148, 0x11150, 0x11173, 0x11174, 0x11176, 0x11177, 0x11180, 0x11182, + 0x11183, 0x111B3, 0x111B6, 0x111BF, 0x111C1, 0x111C5, 0x111C7, 0x111C9, 0x111CD, 0x111CE, + 0x111CF, 0x111D0, 0x111DA, 0x111DB, 0x111DC, 0x111DD, 0x111DE, 0x111E0, 0x11200, 0x11212, + 0x11213, 0x1122C, 0x1122F, 0x11232, 0x11234, 0x11235, 0x11236, 0x11238, 0x1123A, 0x1123B, + 0x1123D, 0x1123E, 0x1123F, 0x11241, 0x11242, 0x11280, 0x11287, 0x11288, 0x11289, 0x1128A, + 0x1128E, 0x1128F, 0x1129E, 0x1129F, 0x112A9, 0x112AA, 0x112B0, 0x112DF, 0x112E0, 0x112E3, + 0x112EB, 0x112F0, 0x112FA, 0x11300, 0x11302, 0x11304, 0x11305, 0x1130D, 0x1130F, 0x11311, + 0x11313, 0x11329, 0x1132A, 0x11331, 0x11332, 0x11334, 0x11335, 0x1133A, 0x1133B, 0x1133D, + 0x1133E, 0x11340, 0x11341, 0x11345, 0x11347, 0x11349, 0x1134B, 0x1134E, 0x11350, 0x11351, + 0x11357, 0x11358, 0x1135D, 0x11362, 0x11364, 0x11366, 0x1136D, 0x11370, 0x11375, 0x11380, + 0x1138A, 0x1138B, 0x1138C, 0x1138E, 0x1138F, 0x11390, 0x113B6, 0x113B7, 0x113B8, 0x113BB, + 0x113C1, 0x113C2, 0x113C3, 0x113C5, 0x113C6, 0x113C7, 0x113CB, 0x113CC, 0x113CE, 0x113CF, + 0x113D0, 0x113D1, 0x113D2, 0x113D3, 0x113D4, 0x113D6, 0x113E1, 0x113E3, 0x11400, 0x11435, + 0x11438, 0x11440, 0x11442, 0x11445, 0x11446, 0x11447, 0x1144B, 0x1144D, 0x11450, 0x1145A, + 0x1145E, 0x1145F, 0x11462, 0x11480, 0x114B0, 0x114B3, 0x114B9, 0x114BA, 0x114BB, 0x114BF, + 0x114C1, 0x114C2, 0x114C4, 0x114C6, 0x114C7, 0x114C8, 0x114D0, 0x114DA, 0x11580, 0x115AF, + 0x115B2, 0x115B6, 0x115B8, 0x115BC, 0x115BE, 0x115BF, 0x115C1, 0x115C2, 0x115C4, 0x115C9, + 0x115D8, 0x115DC, 0x115DE, 0x11600, 0x11630, 0x11633, 0x1163B, 0x1163D, 0x1163E, 0x1163F, + 0x11641, 0x11643, 0x11644, 0x11645, 0x11650, 0x1165A, 0x11680, 0x116AB, 0x116AC, 0x116AD, + 0x116AE, 0x116B0, 0x116B6, 0x116B7, 0x116B8, 0x116B9, 0x116C0, 0x116CA, 0x116D0, 0x116E4, + 0x11700, 0x1171B, 0x1171D, 0x1171E, 0x1171F, 0x11720, 0x11722, 0x11726, 0x11727, 0x1172C, + 0x11730, 0x1173A, 0x1173C, 0x1173F, 0x11740, 0x11747, 0x11800, 0x1182C, 0x1182F, 0x11838, + 0x11839, 0x1183B, 0x118A0, 0x118C0, 0x118E0, 0x118EA, 0x118FF, 0x11907, 0x11909, 0x1190A, + 0x1190C, 0x11914, 0x11915, 0x11917, 0x11918, 0x11930, 0x11936, 0x11937, 0x11939, 0x1193B, + 0x1193D, 0x1193E, 0x1193F, 0x11940, 0x11941, 0x11942, 0x11943, 0x11944, 0x11945, 0x11946, + 0x11947, 0x11950, 0x1195A, 0x119A0, 0x119A8, 0x119AA, 0x119D1, 0x119D4, 0x119D8, 0x119DA, + 0x119DC, 0x119E0, 0x119E1, 0x119E2, 0x119E3, 0x119E4, 0x119E5, 0x11A00, 0x11A01, 0x11A0B, + 0x11A33, 0x11A39, 0x11A3A, 0x11A3B, 0x11A3F, 0x11A42, 0x11A44, 0x11A47, 0x11A48, 0x11A50, + 0x11A51, 0x11A57, 0x11A59, 0x11A5C, 0x11A8A, 0x11A97, 0x11A98, 0x11A9A, 0x11A9B, 0x11A9D, + 0x11A9E, 0x11AB0, 0x11AF9, 0x11B60, 0x11B61, 0x11B62, 0x11B65, 0x11B66, 0x11B67, 0x11B68, + 0x11BC0, 0x11BE1, 0x11BF0, 0x11BFA, 0x11C00, 0x11C09, 0x11C0A, 0x11C2F, 0x11C30, 0x11C37, + 0x11C38, 0x11C3E, 0x11C3F, 0x11C40, 0x11C41, 0x11C43, 0x11C50, 0x11C5A, 0x11C72, 0x11C90, + 0x11C92, 0x11CA8, 0x11CA9, 0x11CAA, 0x11CB1, 0x11CB2, 0x11CB4, 0x11CB5, 0x11CB7, 0x11D00, + 0x11D07, 0x11D08, 0x11D0A, 0x11D0B, 0x11D31, 0x11D37, 0x11D3A, 0x11D3B, 0x11D3C, 0x11D3E, + 0x11D3F, 0x11D46, 0x11D47, 0x11D48, 0x11D50, 0x11D5A, 0x11D60, 0x11D66, 0x11D67, 0x11D69, + 0x11D6A, 0x11D8A, 0x11D8F, 0x11D90, 0x11D92, 0x11D93, 0x11D95, 0x11D96, 0x11D97, 0x11D98, + 0x11D99, 0x11DA0, 0x11DAA, 0x11DB0, 0x11DD9, 0x11DDA, 0x11DDC, 0x11DE0, 0x11DEA, 0x11EE0, + 0x11EF3, 0x11EF5, 0x11EF7, 0x11EF9, 0x11F00, 0x11F02, 0x11F03, 0x11F04, 0x11F11, 0x11F12, + 0x11F34, 0x11F36, 0x11F3B, 0x11F3E, 0x11F40, 0x11F41, 0x11F42, 0x11F43, 0x11F45, 0x11F50, + 0x11F5A, 0x11F5B, 0x11FB0, 0x11FB1, 0x12000, 0x1239A, 0x12400, 0x1246F, 0x12480, 0x12544, + 0x12F90, 0x12FF1, 0x13000, 0x13430, 0x13440, 0x13441, 0x13447, 0x13456, 0x13460, 0x143FB, + 0x14400, 0x14647, 0x16100, 0x1611E, 0x1612A, 0x1612D, 0x16130, 0x1613A, 0x16800, 0x16A39, + 0x16A40, 0x16A5F, 0x16A60, 0x16A6A, 0x16A6E, 0x16A70, 0x16ABF, 0x16AC0, 0x16ACA, 0x16AD0, + 0x16AEE, 0x16AF0, 0x16AF5, 0x16AF6, 0x16B00, 0x16B30, 0x16B37, 0x16B39, 0x16B40, 0x16B44, + 0x16B45, 0x16B50, 0x16B5A, 0x16B63, 0x16B78, 0x16B7D, 0x16B90, 0x16D40, 0x16D43, 0x16D6B, + 0x16D6D, 0x16D6E, 0x16D70, 0x16D7A, 0x16E40, 0x16E60, 0x16E80, 0x16E98, 0x16E99, 0x16EA0, + 0x16EB9, 0x16EBB, 0x16ED4, 0x16F00, 0x16F4B, 0x16F4F, 0x16F50, 0x16F51, 0x16F88, 0x16F8F, + 0x16F93, 0x16FA0, 0x16FE0, 0x16FE2, 0x16FE3, 0x16FE4, 0x16FE5, 0x16FF0, 0x16FF2, 0x16FF4, + 0x16FF7, 0x17000, 0x18CD6, 0x18CFF, 0x18D1F, 0x18D80, 0x18DF3, 0x1AFF0, 0x1AFF4, 0x1AFF5, + 0x1AFFC, 0x1AFFD, 0x1AFFF, 0x1B000, 0x1B123, 0x1B132, 0x1B133, 0x1B150, 0x1B153, 0x1B155, + 0x1B156, 0x1B164, 0x1B168, 0x1B170, 0x1B2FC, 0x1BC00, 0x1BC6B, 0x1BC70, 0x1BC7D, 0x1BC80, + 0x1BC89, 0x1BC90, 0x1BC9A, 0x1BC9D, 0x1BC9F, 0x1BCA0, 0x1BCA4, 0x1CCF0, 0x1CCFA, 0x1CF00, + 0x1CF2E, 0x1CF30, 0x1CF47, 0x1D165, 0x1D167, 0x1D16A, 0x1D16D, 0x1D173, 0x1D17B, 0x1D183, + 0x1D185, 0x1D18C, 0x1D1AA, 0x1D1AE, 0x1D242, 0x1D245, 0x1D400, 0x1D41A, 0x1D434, 0x1D44E, + 0x1D455, 0x1D456, 0x1D468, 0x1D482, 0x1D49C, 0x1D49D, 0x1D49E, 0x1D4A0, 0x1D4A2, 0x1D4A3, + 0x1D4A5, 0x1D4A7, 0x1D4A9, 0x1D4AD, 0x1D4AE, 0x1D4B6, 0x1D4BA, 0x1D4BB, 0x1D4BC, 0x1D4BD, + 0x1D4C4, 0x1D4C5, 0x1D4D0, 0x1D4EA, 0x1D504, 0x1D506, 0x1D507, 0x1D50B, 0x1D50D, 0x1D515, + 0x1D516, 0x1D51D, 0x1D51E, 0x1D538, 0x1D53A, 0x1D53B, 0x1D53F, 0x1D540, 0x1D545, 0x1D546, + 0x1D547, 0x1D54A, 0x1D551, 0x1D552, 0x1D56C, 0x1D586, 0x1D5A0, 0x1D5BA, 0x1D5D4, 0x1D5EE, + 0x1D608, 0x1D622, 0x1D63C, 0x1D656, 0x1D670, 0x1D68A, 0x1D6A6, 0x1D6A8, 0x1D6C1, 0x1D6C2, + 0x1D6DB, 0x1D6DC, 0x1D6E2, 0x1D6FB, 0x1D6FC, 0x1D715, 0x1D716, 0x1D71C, 0x1D735, 0x1D736, + 0x1D74F, 0x1D750, 0x1D756, 0x1D76F, 0x1D770, 0x1D789, 0x1D78A, 0x1D790, 0x1D7A9, 0x1D7AA, + 0x1D7C3, 0x1D7C4, 0x1D7CA, 0x1D7CB, 0x1D7CC, 0x1D7CE, 0x1D800, 0x1DA00, 0x1DA37, 0x1DA3B, + 0x1DA6D, 0x1DA75, 0x1DA76, 0x1DA84, 0x1DA85, 0x1DA88, 0x1DA89, 0x1DA9B, 0x1DAA0, 0x1DAA1, + 0x1DAB0, 0x1DF00, 0x1DF0A, 0x1DF0B, 0x1DF1F, 0x1DF25, 0x1DF2B, 0x1E000, 0x1E007, 0x1E008, + 0x1E019, 0x1E01B, 0x1E022, 0x1E023, 0x1E025, 0x1E026, 0x1E02B, 0x1E030, 0x1E06E, 0x1E08F, + 0x1E090, 0x1E100, 0x1E12D, 0x1E130, 0x1E137, 0x1E13E, 0x1E140, 0x1E14A, 0x1E14E, 0x1E14F, + 0x1E290, 0x1E2AE, 0x1E2AF, 0x1E2C0, 0x1E2EC, 0x1E2F0, 0x1E2FA, 0x1E4D0, 0x1E4EB, 0x1E4EC, + 0x1E4F0, 0x1E4FA, 0x1E5D0, 0x1E5EE, 0x1E5F0, 0x1E5F1, 0x1E5FB, 0x1E6C0, 0x1E6DF, 0x1E6E0, + 0x1E6E3, 0x1E6E4, 0x1E6E6, 0x1E6E7, 0x1E6EE, 0x1E6F0, 0x1E6F5, 0x1E6F6, 0x1E6FE, 0x1E6FF, + 0x1E700, 0x1E7E0, 0x1E7E7, 0x1E7E8, 0x1E7EC, 0x1E7ED, 0x1E7EF, 0x1E7F0, 0x1E7FF, 0x1E800, + 0x1E8C5, 0x1E8D0, 0x1E8D7, 0x1E900, 0x1E922, 0x1E944, 0x1E94B, 0x1E94C, 0x1E950, 0x1E95A, + 0x1EE00, 0x1EE04, 0x1EE05, 0x1EE20, 0x1EE21, 0x1EE23, 0x1EE24, 0x1EE25, 0x1EE27, 0x1EE28, + 0x1EE29, 0x1EE33, 0x1EE34, 0x1EE38, 0x1EE39, 0x1EE3A, 0x1EE3B, 0x1EE3C, 0x1EE42, 0x1EE43, + 0x1EE47, 0x1EE48, 0x1EE49, 0x1EE4A, 0x1EE4B, 0x1EE4C, 0x1EE4D, 0x1EE50, 0x1EE51, 0x1EE53, + 0x1EE54, 0x1EE55, 0x1EE57, 0x1EE58, 0x1EE59, 0x1EE5A, 0x1EE5B, 0x1EE5C, 0x1EE5D, 0x1EE5E, + 0x1EE5F, 0x1EE60, 0x1EE61, 0x1EE63, 0x1EE64, 0x1EE65, 0x1EE67, 0x1EE6B, 0x1EE6C, 0x1EE73, + 0x1EE74, 0x1EE78, 0x1EE79, 0x1EE7D, 0x1EE7E, 0x1EE7F, 0x1EE80, 0x1EE8A, 0x1EE8B, 0x1EE9C, + 0x1EEA1, 0x1EEA4, 0x1EEA5, 0x1EEAA, 0x1EEAB, 0x1EEBC, 0x1F130, 0x1F14A, 0x1F150, 0x1F16A, + 0x1F170, 0x1F18A, 0x1F676, 0x1F679, 0x1FBF0, 0x1FBFA, 0x20000, 0x2A6E0, 0x2A700, 0x2B81E, + 0x2B820, 0x2CEAE, 0x2CEB0, 0x2EBE1, 0x2EBF0, 0x2EE5E, 0x2F800, 0x2FA1E, 0x30000, 0x3134B, + 0x31350, 0x3347A, 0xE0001, 0xE0002, 0xE0020, 0xE0080, 0xE0100, 0xE01F0, + }; + + private static final byte[] VALUES = { + OTHER, SP, LF, SP, CR, OTHER, SP, STERM, CLOSE, OTHER, CLOSE, CLOSE, CLOSE, OTHER, SCONTINUE, SCONTINUE, + ATERM, OTHER, NUMERIC, SCONTINUE, OTHER, STERM, OTHER, UPPER, CLOSE, OTHER, CLOSE, OTHER, LOWER, CLOSE, OTHER, CLOSE, + OTHER, SEP, OTHER, SP, OTHER, LOWER, CLOSE, OTHER, FORMAT, OTHER, LOWER, OTHER, LOWER, CLOSE, OTHER, UPPER, + OTHER, UPPER, LOWER, OTHER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, OLETTER, UPPER, LOWER, OLETTER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, OLETTER, LOWER, LOWER, OLETTER, LOWER, OTHER, OLETTER, OTHER, LOWER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, + UPPER, LOWER, UPPER, LOWER, OLETTER, OTHER, UPPER, LOWER, OTHER, LOWER, LOWER, SCONTINUE, UPPER, OTHER, UPPER, OTHER, + UPPER, OTHER, UPPER, OTHER, UPPER, LOWER, UPPER, OTHER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, OTHER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, OTHER, EXTEND, EXTEND, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, OTHER, UPPER, OTHER, OLETTER, OTHER, SCONTINUE, OTHER, LOWER, + STERM, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, OLETTER, OTHER, OLETTER, OLETTER, + OTHER, NUMERIC, OTHER, SCONTINUE, OTHER, EXTEND, OTHER, FORMAT, STERM, OLETTER, OLETTER, OLETTER, EXTEND, NUMERIC, OTHER, NUMERIC, + OTHER, OLETTER, EXTEND, OLETTER, STERM, OLETTER, EXTEND, NUMERIC, OTHER, EXTEND, OLETTER, EXTEND, OTHER, EXTEND, OLETTER, NUMERIC, + OLETTER, OTHER, OLETTER, STERM, OTHER, FORMAT, OLETTER, EXTEND, OLETTER, EXTEND, OTHER, OLETTER, EXTEND, OLETTER, OTHER, NUMERIC, + OLETTER, EXTEND, OLETTER, OTHER, SCONTINUE, STERM, OLETTER, OTHER, EXTEND, OTHER, OLETTER, EXTEND, OLETTER, EXTEND, OLETTER, EXTEND, + OLETTER, EXTEND, OTHER, STERM, OTHER, STERM, OTHER, STERM, OTHER, OLETTER, EXTEND, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, NUMERIC, OTHER, EXTEND, OLETTER, OLETTER, EXTEND, NUMERIC, EXTEND, EXTEND, OLETTER, EXTEND, EXTEND, EXTEND, OLETTER, EXTEND, + EXTEND, EXTEND, EXTEND, EXTEND, OLETTER, EXTEND, OLETTER, EXTEND, STERM, NUMERIC, OTHER, OLETTER, OLETTER, EXTEND, EXTEND, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, OLETTER, EXTEND, EXTEND, + OTHER, EXTEND, OTHER, EXTEND, EXTEND, OLETTER, OTHER, EXTEND, OTHER, OLETTER, OTHER, OLETTER, EXTEND, OTHER, NUMERIC, OLETTER, + OTHER, OLETTER, OTHER, EXTEND, OTHER, EXTEND, EXTEND, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, OTHER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, + OTHER, OLETTER, OTHER, OLETTER, OTHER, NUMERIC, EXTEND, OLETTER, EXTEND, OTHER, EXTEND, EXTEND, OTHER, OLETTER, OTHER, OLETTER, + OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, OLETTER, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, + OTHER, EXTEND, EXTEND, OTHER, OLETTER, OTHER, OLETTER, EXTEND, OTHER, NUMERIC, OTHER, OLETTER, EXTEND, OTHER, EXTEND, EXTEND, + OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, OLETTER, EXTEND, + EXTEND, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, OTHER, OLETTER, OTHER, OLETTER, EXTEND, + OTHER, NUMERIC, OTHER, OLETTER, OTHER, EXTEND, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, OTHER, + EXTEND, EXTEND, OTHER, OLETTER, OTHER, EXTEND, OTHER, NUMERIC, OTHER, EXTEND, EXTEND, EXTEND, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, OTHER, EXTEND, OLETTER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, OLETTER, + OTHER, OLETTER, OTHER, OLETTER, EXTEND, OTHER, NUMERIC, OTHER, OLETTER, EXTEND, EXTEND, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, OLETTER, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, OTHER, EXTEND, + EXTEND, OTHER, EXTEND, OTHER, OLETTER, OTHER, OLETTER, EXTEND, OTHER, NUMERIC, OTHER, OLETTER, EXTEND, OTHER, EXTEND, EXTEND, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, EXTEND, OLETTER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, OLETTER, OTHER, + OLETTER, EXTEND, OTHER, OLETTER, EXTEND, OTHER, NUMERIC, OTHER, OLETTER, OTHER, EXTEND, EXTEND, OTHER, OLETTER, OTHER, OLETTER, + OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, OTHER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, + NUMERIC, OTHER, EXTEND, OTHER, OLETTER, EXTEND, OLETTER, EXTEND, OTHER, OLETTER, OLETTER, EXTEND, OTHER, NUMERIC, OTHER, OLETTER, + OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, EXTEND, OLETTER, EXTEND, OLETTER, OTHER, OLETTER, + OTHER, OLETTER, OTHER, EXTEND, OTHER, NUMERIC, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, OTHER, NUMERIC, OTHER, EXTEND, + OTHER, EXTEND, OTHER, EXTEND, CLOSE, CLOSE, CLOSE, CLOSE, EXTEND, OLETTER, OTHER, OLETTER, OTHER, EXTEND, EXTEND, EXTEND, + OTHER, EXTEND, OLETTER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, + EXTEND, EXTEND, OLETTER, NUMERIC, STERM, OTHER, OLETTER, EXTEND, EXTEND, OLETTER, EXTEND, OLETTER, EXTEND, OLETTER, EXTEND, OLETTER, + EXTEND, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OLETTER, EXTEND, NUMERIC, EXTEND, EXTEND, OTHER, UPPER, OTHER, UPPER, + OTHER, UPPER, OTHER, OLETTER, OTHER, LOWER, OLETTER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, + OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, + OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, OTHER, STERM, OTHER, STERM, OTHER, OLETTER, OTHER, UPPER, + OTHER, LOWER, OTHER, OLETTER, OTHER, STERM, OLETTER, SP, OLETTER, CLOSE, CLOSE, OTHER, OLETTER, OTHER, OLETTER, OLETTER, + OTHER, OLETTER, EXTEND, EXTEND, OTHER, OLETTER, EXTEND, EXTEND, STERM, OTHER, OLETTER, EXTEND, OTHER, OLETTER, OTHER, OLETTER, + OTHER, EXTEND, OTHER, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, STERM, OTHER, OLETTER, OTHER, OLETTER, + EXTEND, OTHER, NUMERIC, OTHER, SCONTINUE, STERM, OTHER, SCONTINUE, STERM, OTHER, EXTEND, FORMAT, EXTEND, NUMERIC, OTHER, OLETTER, + OLETTER, OLETTER, OTHER, OLETTER, EXTEND, OLETTER, EXTEND, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, EXTEND, EXTEND, + EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, STERM, NUMERIC, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, + OTHER, NUMERIC, NUMERIC, OTHER, OLETTER, EXTEND, EXTEND, EXTEND, OTHER, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, + EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, NUMERIC, OTHER, NUMERIC, OTHER, OLETTER, STERM, OTHER, EXTEND, + EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, + OLETTER, OTHER, STERM, NUMERIC, STERM, OTHER, STERM, OTHER, EXTEND, OTHER, STERM, EXTEND, EXTEND, OLETTER, EXTEND, EXTEND, + EXTEND, EXTEND, EXTEND, EXTEND, OLETTER, NUMERIC, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, + OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, STERM, OTHER, NUMERIC, OTHER, OLETTER, NUMERIC, OLETTER, OLETTER, STERM, LOWER, + UPPER, LOWER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, OLETTER, EXTEND, OLETTER, EXTEND, + OLETTER, EXTEND, EXTEND, OLETTER, OTHER, LOWER, LOWER, LOWER, LOWER, LOWER, LOWER, EXTEND, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, OTHER, UPPER, OTHER, LOWER, UPPER, LOWER, UPPER, LOWER, OTHER, UPPER, + OTHER, LOWER, OTHER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, UPPER, LOWER, UPPER, LOWER, OTHER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, OTHER, LOWER, UPPER, OTHER, LOWER, OTHER, LOWER, OTHER, LOWER, UPPER, OTHER, + LOWER, OTHER, LOWER, UPPER, OTHER, LOWER, UPPER, OTHER, LOWER, OTHER, LOWER, UPPER, OTHER, SP, FORMAT, EXTEND, + FORMAT, OTHER, SCONTINUE, OTHER, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, OTHER, ATERM, OTHER, SEP, SEP, + FORMAT, SP, OTHER, CLOSE, CLOSE, OTHER, STERM, OTHER, CLOSE, CLOSE, STERM, OTHER, SP, FORMAT, OTHER, FORMAT, + OTHER, LOWER, OTHER, CLOSE, CLOSE, LOWER, OTHER, CLOSE, CLOSE, OTHER, LOWER, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, + EXTEND, OTHER, UPPER, OTHER, UPPER, OTHER, LOWER, UPPER, LOWER, UPPER, LOWER, OTHER, UPPER, OTHER, UPPER, OTHER, + UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, LOWER, UPPER, LOWER, OLETTER, LOWER, OTHER, LOWER, UPPER, + OTHER, UPPER, LOWER, OTHER, LOWER, OTHER, UPPER, LOWER, OLETTER, UPPER, LOWER, OLETTER, OTHER, CLOSE, CLOSE, CLOSE, + CLOSE, OTHER, CLOSE, CLOSE, OTHER, UPPER, LOWER, OTHER, CLOSE, OTHER, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, + CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, OTHER, CLOSE, CLOSE, OTHER, CLOSE, CLOSE, CLOSE, CLOSE, + CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, OTHER, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, + CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, OTHER, CLOSE, CLOSE, + CLOSE, CLOSE, OTHER, CLOSE, CLOSE, OTHER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, OTHER, UPPER, LOWER, + UPPER, LOWER, EXTEND, UPPER, LOWER, OTHER, STERM, OTHER, LOWER, OTHER, LOWER, OTHER, LOWER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, EXTEND, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, + OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, + CLOSE, OTHER, CLOSE, CLOSE, OTHER, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, OTHER, + STERM, OLETTER, OTHER, STERM, OTHER, CLOSE, OTHER, STERM, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, + OTHER, SP, SCONTINUE, STERM, OTHER, OLETTER, OLETTER, OLETTER, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, + CLOSE, CLOSE, OTHER, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, OTHER, CLOSE, CLOSE, OTHER, OLETTER, + EXTEND, EXTEND, OTHER, OLETTER, OTHER, OLETTER, OLETTER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, OTHER, OLETTER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, + OLETTER, OLETTER, OTHER, OLETTER, OLETTER, OTHER, STERM, OLETTER, OLETTER, OTHER, STERM, OLETTER, NUMERIC, OLETTER, OTHER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, OLETTER, EXTEND, EXTEND, + OTHER, EXTEND, OTHER, OLETTER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + LOWER, EXTEND, OLETTER, OLETTER, EXTEND, OTHER, STERM, OTHER, STERM, OTHER, OLETTER, OTHER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, LOWER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, OLETTER, OTHER, UPPER, LOWER, UPPER, LOWER, OLETTER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, + LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, OTHER, LOWER, + UPPER, LOWER, OLETTER, LOWER, LOWER, OLETTER, EXTEND, OLETTER, EXTEND, OLETTER, EXTEND, OLETTER, EXTEND, EXTEND, EXTEND, OTHER, + EXTEND, OTHER, OLETTER, OTHER, STERM, OTHER, EXTEND, OLETTER, EXTEND, EXTEND, OTHER, STERM, NUMERIC, OTHER, EXTEND, OLETTER, + OTHER, OLETTER, OTHER, OLETTER, EXTEND, NUMERIC, OLETTER, EXTEND, OTHER, STERM, OLETTER, EXTEND, EXTEND, OTHER, OLETTER, OTHER, + EXTEND, EXTEND, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, STERM, OTHER, OLETTER, NUMERIC, OTHER, OLETTER, + EXTEND, OLETTER, OLETTER, NUMERIC, OLETTER, OTHER, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, OLETTER, EXTEND, OLETTER, + EXTEND, EXTEND, OTHER, NUMERIC, OTHER, STERM, OLETTER, OLETTER, OLETTER, OTHER, OLETTER, EXTEND, EXTEND, EXTEND, OLETTER, EXTEND, + OLETTER, EXTEND, OLETTER, EXTEND, OLETTER, EXTEND, OLETTER, EXTEND, OLETTER, OTHER, OLETTER, OLETTER, OTHER, OLETTER, EXTEND, EXTEND, + EXTEND, STERM, OLETTER, OLETTER, EXTEND, EXTEND, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, + OTHER, LOWER, OTHER, LOWER, LOWER, LOWER, OTHER, LOWER, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, STERM, EXTEND, + EXTEND, OTHER, NUMERIC, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, LOWER, OTHER, + LOWER, OTHER, OLETTER, EXTEND, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, CLOSE, CLOSE, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, SCONTINUE, STERM, SCONTINUE, + STERM, CLOSE, CLOSE, OTHER, EXTEND, OTHER, SCONTINUE, OTHER, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, + CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, OTHER, CLOSE, CLOSE, OTHER, SCONTINUE, ATERM, OTHER, SCONTINUE, + STERM, SCONTINUE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, CLOSE, OTHER, SCONTINUE, OTHER, OLETTER, OTHER, OLETTER, OTHER, FORMAT, + OTHER, STERM, OTHER, CLOSE, CLOSE, OTHER, SCONTINUE, SCONTINUE, ATERM, OTHER, NUMERIC, SCONTINUE, OTHER, STERM, OTHER, UPPER, + CLOSE, OTHER, CLOSE, OTHER, LOWER, CLOSE, OTHER, CLOSE, OTHER, CLOSE, CLOSE, STERM, CLOSE, CLOSE, SCONTINUE, OTHER, + OLETTER, OLETTER, OLETTER, EXTEND, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, FORMAT, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + EXTEND, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, OTHER, OLETTER, OTHER, OLETTER, OLETTER, OLETTER, OLETTER, OTHER, OLETTER, + EXTEND, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, UPPER, LOWER, OLETTER, OTHER, NUMERIC, OTHER, + UPPER, OTHER, LOWER, OTHER, OLETTER, OTHER, OLETTER, OTHER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, + LOWER, OTHER, LOWER, OTHER, LOWER, OTHER, LOWER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + LOWER, OLETTER, LOWER, OTHER, LOWER, OTHER, LOWER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, EXTEND, OTHER, EXTEND, OTHER, STERM, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, EXTEND, + OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, UPPER, OTHER, LOWER, OTHER, OLETTER, + EXTEND, OTHER, NUMERIC, OTHER, NUMERIC, OLETTER, OLETTER, OLETTER, UPPER, OTHER, EXTEND, OTHER, OLETTER, LOWER, OTHER, OLETTER, + OTHER, EXTEND, OTHER, OLETTER, OTHER, OLETTER, OLETTER, OLETTER, OTHER, EXTEND, OLETTER, OTHER, OLETTER, OTHER, OLETTER, EXTEND, + OTHER, STERM, OTHER, OLETTER, EXTEND, STERM, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, EXTEND, EXTEND, OLETTER, EXTEND, + STERM, OTHER, NUMERIC, EXTEND, OLETTER, EXTEND, OLETTER, OTHER, EXTEND, EXTEND, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, + NUMERIC, STERM, EXTEND, OTHER, NUMERIC, OTHER, OLETTER, OTHER, NUMERIC, OTHER, EXTEND, OLETTER, EXTEND, EXTEND, EXTEND, OTHER, + NUMERIC, OTHER, STERM, OLETTER, EXTEND, OLETTER, OTHER, OLETTER, EXTEND, OTHER, OLETTER, OTHER, EXTEND, EXTEND, OLETTER, EXTEND, + EXTEND, EXTEND, OLETTER, STERM, OTHER, EXTEND, STERM, EXTEND, EXTEND, NUMERIC, OLETTER, OTHER, OLETTER, OTHER, STERM, OTHER, + OLETTER, OTHER, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, STERM, OTHER, STERM, OTHER, EXTEND, OLETTER, EXTEND, + OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, STERM, OTHER, OLETTER, EXTEND, EXTEND, EXTEND, + OTHER, NUMERIC, OTHER, EXTEND, EXTEND, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, EXTEND, OLETTER, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, OLETTER, OTHER, EXTEND, OTHER, + OLETTER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, + EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, OLETTER, EXTEND, OLETTER, + STERM, OTHER, EXTEND, OTHER, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OLETTER, STERM, OTHER, NUMERIC, OTHER, + EXTEND, OLETTER, OTHER, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OLETTER, OTHER, OLETTER, OTHER, + NUMERIC, OTHER, OLETTER, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, STERM, OTHER, STERM, OLETTER, EXTEND, + OTHER, OLETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, STERM, OTHER, OLETTER, OTHER, NUMERIC, OTHER, OLETTER, EXTEND, + EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OLETTER, OTHER, NUMERIC, OTHER, NUMERIC, OTHER, OLETTER, OTHER, EXTEND, EXTEND, + EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, NUMERIC, OTHER, STERM, OTHER, OLETTER, OTHER, OLETTER, EXTEND, EXTEND, EXTEND, + EXTEND, OTHER, UPPER, LOWER, NUMERIC, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, EXTEND, + OTHER, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, OLETTER, EXTEND, OLETTER, EXTEND, EXTEND, STERM, OTHER, STERM, OTHER, NUMERIC, + OTHER, OLETTER, OTHER, OLETTER, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, OLETTER, OTHER, OLETTER, EXTEND, OTHER, OLETTER, + EXTEND, OLETTER, EXTEND, EXTEND, OLETTER, EXTEND, OTHER, STERM, OTHER, EXTEND, OTHER, OLETTER, EXTEND, EXTEND, EXTEND, OLETTER, + EXTEND, EXTEND, EXTEND, OTHER, STERM, OLETTER, OTHER, OLETTER, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, + OLETTER, OTHER, NUMERIC, OTHER, OLETTER, OTHER, OLETTER, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, OLETTER, STERM, OTHER, + NUMERIC, OTHER, OLETTER, OTHER, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, OLETTER, OTHER, OLETTER, + OTHER, OLETTER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OLETTER, EXTEND, OTHER, NUMERIC, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, OLETTER, OTHER, NUMERIC, OTHER, OLETTER, + OLETTER, OLETTER, OTHER, NUMERIC, OTHER, OLETTER, EXTEND, EXTEND, STERM, OTHER, EXTEND, OLETTER, EXTEND, OLETTER, OTHER, OLETTER, + EXTEND, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, STERM, OTHER, NUMERIC, EXTEND, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, FORMAT, EXTEND, OLETTER, EXTEND, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, EXTEND, EXTEND, EXTEND, NUMERIC, OTHER, OLETTER, OTHER, OLETTER, OTHER, NUMERIC, OTHER, STERM, OLETTER, OTHER, NUMERIC, + OTHER, OLETTER, OTHER, EXTEND, STERM, OTHER, OLETTER, EXTEND, STERM, OTHER, OLETTER, STERM, OTHER, NUMERIC, OTHER, OLETTER, + OTHER, OLETTER, OTHER, OLETTER, OLETTER, OLETTER, OTHER, STERM, NUMERIC, OTHER, UPPER, LOWER, OTHER, STERM, OTHER, UPPER, + OTHER, LOWER, OTHER, OLETTER, OTHER, EXTEND, OLETTER, EXTEND, OTHER, EXTEND, OLETTER, OTHER, OLETTER, OTHER, OLETTER, EXTEND, + OTHER, EXTEND, OLETTER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, + OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, + OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, STERM, FORMAT, OTHER, NUMERIC, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, + EXTEND, OTHER, EXTEND, FORMAT, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, UPPER, LOWER, UPPER, LOWER, + OTHER, LOWER, UPPER, LOWER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, UPPER, LOWER, + OTHER, LOWER, OTHER, LOWER, OTHER, LOWER, UPPER, LOWER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, + LOWER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, LOWER, UPPER, LOWER, UPPER, LOWER, + UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, UPPER, LOWER, OTHER, UPPER, OTHER, LOWER, OTHER, LOWER, UPPER, OTHER, + LOWER, OTHER, LOWER, UPPER, OTHER, LOWER, OTHER, LOWER, UPPER, OTHER, LOWER, OTHER, LOWER, UPPER, OTHER, LOWER, + OTHER, LOWER, UPPER, LOWER, OTHER, NUMERIC, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, STERM, + OTHER, EXTEND, OTHER, EXTEND, OTHER, LOWER, OLETTER, LOWER, OTHER, LOWER, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, + OTHER, EXTEND, OTHER, EXTEND, OTHER, LOWER, OTHER, EXTEND, OTHER, OLETTER, OTHER, EXTEND, OLETTER, OTHER, NUMERIC, OTHER, + OLETTER, OTHER, OLETTER, EXTEND, OTHER, OLETTER, EXTEND, NUMERIC, OTHER, OLETTER, OLETTER, EXTEND, NUMERIC, OTHER, OLETTER, EXTEND, + OLETTER, NUMERIC, OTHER, OLETTER, OTHER, OLETTER, EXTEND, OLETTER, EXTEND, OLETTER, EXTEND, OLETTER, EXTEND, OTHER, OLETTER, OLETTER, + OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, EXTEND, OTHER, UPPER, LOWER, EXTEND, + OLETTER, OTHER, NUMERIC, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, UPPER, OTHER, UPPER, OTHER, UPPER, OTHER, CLOSE, OTHER, NUMERIC, OTHER, + OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, OLETTER, OTHER, + FORMAT, OTHER, EXTEND, OTHER, EXTEND, OTHER, + }; + + static byte propertyOf(int codePoint) { + int low = 0; + int high = STARTS.length - 1; + while (low <= high) { + int middle = (low + high) >>> 1; + if (STARTS[middle] <= codePoint) { + low = middle + 1; + } else { + high = middle - 1; + } + } + return VALUES[Math.max(0, high)]; + } + + private SentenceBreakData() { + } +} diff --git a/third_party/joni/src/org/joni/constants/internal/AnchorType.java b/third_party/joni/src/org/joni/constants/internal/AnchorType.java index 6b3c845f09..e329d0cf29 100644 --- a/third_party/joni/src/org/joni/constants/internal/AnchorType.java +++ b/third_party/joni/src/org/joni/constants/internal/AnchorType.java @@ -45,6 +45,8 @@ public interface AnchorType { int KEEP = (1<<16); int GRAPHEME_BOUNDARY = (1<<17); int NOT_GRAPHEME_BOUNDARY = (1<<18); + int SENTENCE_BOUNDARY = (1<<19); + int NOT_SENTENCE_BOUNDARY = (1<<20); int ALLOWED_IN_LB = ( LOOK_BEHIND | LOOK_BEHIND_NOT | @@ -57,6 +59,8 @@ public interface AnchorType { NOT_WORD_BOUND | GRAPHEME_BOUNDARY | NOT_GRAPHEME_BOUNDARY | + SENTENCE_BOUNDARY | + NOT_SENTENCE_BOUNDARY | WORD_BEGIN | WORD_END ); @@ -72,6 +76,8 @@ public interface AnchorType { NOT_WORD_BOUND | GRAPHEME_BOUNDARY | NOT_GRAPHEME_BOUNDARY | + SENTENCE_BOUNDARY | + NOT_SENTENCE_BOUNDARY | WORD_BEGIN | WORD_END ); } diff --git a/third_party/joni/src/org/joni/constants/internal/OPCode.java b/third_party/joni/src/org/joni/constants/internal/OPCode.java index 05a52caa63..7b649ab2dd 100644 --- a/third_party/joni/src/org/joni/constants/internal/OPCode.java +++ b/third_party/joni/src/org/joni/constants/internal/OPCode.java @@ -156,6 +156,8 @@ public interface OPCode { int CHECK_LOOK_BEHIND_END = 112; /* require current position at saved negative boundary */ int GRAPHEME_BOUNDARY = 113; /* Perl \b{gcb} */ int NOT_GRAPHEME_BOUNDARY = 114; /* Perl \B{gcb} */ + int SENTENCE_BOUNDARY = 115; /* Perl \b{sb} */ + int NOT_SENTENCE_BOUNDARY = 116; /* Perl \B{sb} */ String[] OpCodeNames = Config.DEBUG_COMPILE ? new String[] { "finish", /*OP_FINISH*/ @@ -274,6 +276,8 @@ public interface OPCode { "check-look-behind-end", /*OP_CHECK_LOOK_BEHIND_END*/ "grapheme-boundary", /*OP_GRAPHEME_BOUNDARY*/ "not-grapheme-boundary", /*OP_NOT_GRAPHEME_BOUNDARY*/ + "sentence-boundary", /*OP_SENTENCE_BOUNDARY*/ + "not-sentence-boundary", /*OP_NOT_SENTENCE_BOUNDARY*/ } : null; int[] OpCodeArgTypes = Config.DEBUG_COMPILE ? new int[] { @@ -393,5 +397,7 @@ public interface OPCode { Arguments.NON, /*OP_CHECK_LOOK_BEHIND_END*/ Arguments.NON, /*OP_GRAPHEME_BOUNDARY*/ Arguments.NON, /*OP_NOT_GRAPHEME_BOUNDARY*/ + Arguments.NON, /*OP_SENTENCE_BOUNDARY*/ + Arguments.NON, /*OP_NOT_SENTENCE_BOUNDARY*/ } : null; } diff --git a/third_party/joni/test/org/joni/test/TestPerlSentenceBoundary.java b/third_party/joni/test/org/joni/test/TestPerlSentenceBoundary.java new file mode 100644 index 0000000000..0ac3abb86a --- /dev/null +++ b/third_party/joni/test/org/joni/test/TestPerlSentenceBoundary.java @@ -0,0 +1,65 @@ +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni.test; + +import static org.junit.Assert.assertEquals; + +import java.nio.charset.StandardCharsets; + +import org.jcodings.specific.UTF8Encoding; +import org.joni.Option; +import org.joni.Regex; +import org.joni.Syntax; +import org.junit.Test; + +public class TestPerlSentenceBoundary { + private static int search(String pattern, String input) { + byte[] patternBytes = pattern.getBytes(StandardCharsets.UTF_8); + byte[] inputBytes = input.getBytes(StandardCharsets.UTF_8); + Regex regex = new Regex(patternBytes, 0, patternBytes.length, Option.NONE, + UTF8Encoding.INSTANCE, Syntax.PerlNG); + return regex.matcher(inputBytes).search(0, inputBytes.length, Option.NONE); + } + + private static void assertBoundary(String left, String right) { + assertEquals(0, search("\\A" + left + "\\b{sb}" + right + "\\z", left + right)); + } + + private static void assertNoBoundary(String left, String right) { + assertEquals(0, search("\\A" + left + "\\B{sb}" + right + "\\z", left + right)); + } + + @Test + public void implementsUnicodeSentenceBreakRules() { + assertEquals(0, search("\\A\\r\\B{sb}\\n\\z", "\r\n")); + assertBoundary("\r", "A"); + assertNoBoundary("A", "̈"); + assertNoBoundary(".", "1"); + assertNoBoundary("A.", "B"); + assertNoBoundary("A.” ", "a"); + assertNoBoundary("etc.”’ ", "‘“the"); + assertNoBoundary("A.", "”"); + assertBoundary("A. ", "B"); + assertBoundary("A!", "B"); + assertBoundary("A!", "a"); + assertNoBoundary("A.\n̈", "B"); + assertNoBoundary("A", "B"); + } +} From 10afbbec8e1549f42ee3021f767745c70bc8adb0 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:19 +0200 Subject: [PATCH 23/60] feat(regex): add native Perl word boundaries (#1028) Squashed integration of PR #1028 at 52838433f. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 49 +- dev/tools/generate_joni_word_break_data.pl | 153 +++++ docs/reference/feature-matrix.md | 5 +- .../unit/regex/word_boundary_rules.t | 57 ++ .../joni/src/org/joni/ArrayCompiler.java | 8 + .../joni/src/org/joni/ByteCodeMachine.java | 132 ++++ third_party/joni/src/org/joni/Lexer.java | 6 + third_party/joni/src/org/joni/Parser.java | 76 ++- .../joni/src/org/joni/WordBreakData.java | 567 ++++++++++++++++++ .../joni/constants/internal/AnchorType.java | 6 + .../org/joni/constants/internal/OPCode.java | 6 + .../src/org/joni/exception/ErrorMessages.java | 7 + .../joni/test/org/joni/test/TestPerl.java | 17 + .../org/joni/test/TestPerlWordBoundary.java | 67 +++ 14 files changed, 1134 insertions(+), 22 deletions(-) create mode 100644 dev/tools/generate_joni_word_break_data.pl create mode 100644 src/test/resources/unit/regex/word_boundary_rules.t create mode 100644 third_party/joni/src/org/joni/WordBreakData.java create mode 100644 third_party/joni/test/org/joni/test/TestPerlWordBoundary.java diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 386164b01c..7daa1b7196 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,15 +224,15 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1026, stacked -on draft PR #1025 and review-ready PR #1024. It includes the completed callback/runtime slices, +The published integration stack is preserved through draft PR #1027, stacked +on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the first 524 lines of retired Java-only preprocessor code. Every published slice -has a warning-free combined `make` checkpoint. The current WIP adds native -sentence boundaries above that checkpoint; independently validated alpha -assertion aliases and global zero-width `/g` progression remain parallel -integration slices. +has a warning-free combined `make` checkpoint. PR #1027 adds native sentence +boundaries. The current WIP integrates independently validated alpha assertion +aliases and native word boundaries, while global zero-width `/g` progression +and line boundaries remain parallel integration slices. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -261,8 +261,13 @@ remaining sentence assertions without coupling the Joni fork to ICU or the PerlOnJava runtime. The most recent exact property chunks 01–04 remain 98,092/167,501 on both -execution backends. Combined with the complete boundary chunk 05 and unchanged -chunks 06–10, current generated evidence is 113,068/407,367. A resource-contended +execution backends. Native Joni word assertions implement WB1–WB16 and WB999 +from repository-pinned Perl 5.44 Unicode 17.0 Word_Break and +Extended_Pictographic data. The 33-assertion focused oracle passes on system +Perl, JVM, and interpreter, direct Joni exercises the same path, and generated +chunk 10 passes 19,510/19,510 identically on both execution backends. Combined +with the complete boundary chunk 05 and unchanged chunks 06–09, current +generated evidence is 132,578/407,367. A resource-contended current-head refresh did not reproduce a complete exact JVM/interpreter pair, so it does not replace that accepted baseline. @@ -454,6 +459,10 @@ matcher-specific timeouts on both execution backends. - [x] Integrated native braced-octal parsing and missing-close/empty diagnostics plus fatal unterminated braced-hex diagnostics (`55433291a`, `913e2b583`) with exact JVM/interpreter `reg_mesg.t` parity. + - [x] Integrated native Joni alpha assertion aliases `pla`, `plb`, `nla`, + `nlb`, and `atomic` (`a6255fbff`, integrated as `49d7d9648`). The focused + 25-case oracle passes on both execution backends and the generated alpha + corpus gains 98 passing assertions per backend with zero regressions. - [x] Fixed byte-mode substitution of upgraded marker regexes so chunks 05–10 exercise real boundary subjects with exact JVM/interpreter plans. - [x] Implemented native Joni GCB assertions for GB1–GB13 and GB999 and aligned @@ -462,7 +471,11 @@ matcher-specific timeouts on both execution backends. - [x] Implemented native Joni sentence assertions for SB1–SB11 and SB998 from a reproducible Perl 5.44 Unicode 17.0 table. The focused oracle passes 23/23 and generated chunk 05 passes 14,976/14,976 on both execution backends. - - [ ] Implement native Joni line and word boundary algorithms, then + - [x] Implemented native Joni word assertions for WB1–WB16 and WB999 from + reproducible Perl 5.44 Unicode 17.0 Word_Break and Extended_Pictographic + tables. The focused oracle passes 33/33 and generated chunk 10 passes + 19,510/19,510 on both execution backends. + - [ ] Implement the native Joni line boundary algorithm, then close the remaining property and boundary failures before marking Phase 3 complete. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate @@ -506,22 +519,20 @@ matcher-specific timeouts on both execution backends. ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1026, then publish the - sentence-boundary slice as the next stacked PR. Integrate the independently - validated alpha-assertion alias commit and the pending zero-width `/g` - adapter commit after their non-overlapping gates are complete. -2. Continue boundary semantics natively in Joni with line and word breaks - (0/224,890 in chunks 06–10). PerlOnJava4 owns line-boundary preparation while - the coordinator takes word boundaries after the shared sentence base is - published. Use pinned Perl 5.44 Unicode data and +1. Land review-ready PR #1024 and draft PRs #1025–#1027. Publish the current + WIP integration branch containing alpha-assertion aliases, then integrate the + pending zero-width `/g` adapter commit after its non-overlapping gates are + complete. +2. Continue boundary semantics natively in Joni with line breaks + (0/205,380 in chunks 06–09). PerlOnJava4 owns the non-overlapping line slice; + word boundaries are complete at 19,510/19,510 in chunk 10. Use pinned Perl 5.44 Unicode data and the generated chunks as the release oracle; remove each simplified Java boundary rewrite only after native parity. 3. Implement the remaining property clusters in measured order: Block, Script/Script_Extensions, Numeric_Value, Joining_Group, General_Category, break-property values, and Age/In/Present_In. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. - Integrate the completed alpha assertion aliases and close underscored numeric - escapes with focused standard-Perl gates. + Close underscored numeric escapes with focused standard-Perl gates. 4. Rerun the forced-Joni 80-file corpus on JVM and interpreter from the combined head. Save complete JSON and logs, publish the missing differential report, and compare every file with both the Phase 0 result and PR 958 under the diff --git a/dev/tools/generate_joni_word_break_data.pl b/dev/tools/generate_joni_word_break_data.pl new file mode 100644 index 0000000000..62c52c411c --- /dev/null +++ b/dev/tools/generate_joni_word_break_data.pl @@ -0,0 +1,153 @@ +#!/usr/bin/env perl +use strict; +use warnings; + +my $unicode_root = 'perl5/lib/unicore'; +my $expected_unicode_version = '17.0.0'; +open my $version_fh, '<', "$unicode_root/version" or die "Can't read Unicode version: $!\n"; +chomp(my $unicode_version = <$version_fh>); +close $version_fh; +die "Expected Unicode $expected_unicode_version, found $unicode_version\n" + unless $unicode_version eq $expected_unicode_version; + +sub read_property_ranges { + my ($path, $wanted_property) = @_; + open my $fh, '<', $path or die "Can't read $path: $!\n"; + my @ranges; + while (<$fh>) { + next unless /^([0-9A-F]+)(?:\.\.([0-9A-F]+))?\s*;\s*([A-Za-z_]+)/; + my ($start, $end, $property) = (hex($1), defined($2) ? hex($2) : hex($1), $3); + next if defined($wanted_property) && $property ne $wanted_property; + push @ranges, [$start, $end, $property]; + } + close $fh; + return sort { $a->[0] <=> $b->[0] } @ranges; +} + +my @word_ranges = read_property_ranges("$unicode_root/auxiliary/WordBreakProperty.txt"); +my (@starts, @values); +my $cursor = 0; +for my $range (@word_ranges) { + my ($start, $end, $property) = @$range; + die "Overlapping Word_Break ranges at U+", sprintf('%04X', $start), "\n" + if $start < $cursor; + if ($start > $cursor) { + push @starts, $cursor; + push @values, 'Other'; + } + push @starts, $start; + push @values, $property; + $cursor = $end + 1; +} +if ($cursor <= 0x10ffff) { + push @starts, $cursor; + push @values, 'Other'; +} + +my @pictographic_ranges = read_property_ranges( + "$unicode_root/emoji/emoji.txt", 'Extended_Pictographic'); + +my @classes = qw( + Other ALetter CR Double_Quote Extend ExtendNumLet Format Hebrew_Letter + Katakana LF MidLetter MidNum MidNumLet Newline Numeric Regional_Indicator + Single_Quote WSegSpace ZWJ +); +my %class_id; +@class_id{@classes} = (0 .. $#classes); + +for my $value (@values) { + die "Unknown Word_Break value '$value'\n" unless exists $class_id{$value}; +} + +sub constant_name { + my ($name) = @_; + $name =~ s/([a-z])([A-Z])/$1_$2/g; + return uc $name; +} + +print <<'HEADER'; +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni; + +// Generated by dev/tools/generate_joni_word_break_data.pl. Do not edit manually. +final class WordBreakData { +HEADER + +print " static final String UNICODE_VERSION = \"$unicode_version\";\n"; +for my $i (0 .. $#classes) { + print " static final byte ", constant_name($classes[$i]), " = $i;\n"; +} + +print "\n private static final int[] STARTS = {\n"; +for (my $i = 0; $i < @starts; $i += 10) { + my $end = $i + 9 < $#starts ? $i + 9 : $#starts; + print " ", join(', ', map { sprintf '0x%X', $starts[$_] } $i .. $end), ",\n"; +} +print " };\n\n private static final byte[] VALUES = {\n"; +for (my $i = 0; $i < @values; $i += 16) { + my $end = $i + 15 < $#values ? $i + 15 : $#values; + print " ", join(', ', map { constant_name($values[$_]) } $i .. $end), ",\n"; +} +print " };\n\n private static final int[] EXTENDED_PICTOGRAPHIC_RANGES = {\n"; +my @pictographic_pairs = map { ($_->[0], $_->[1]) } @pictographic_ranges; +for (my $i = 0; $i < @pictographic_pairs; $i += 8) { + my $end = $i + 7 < $#pictographic_pairs ? $i + 7 : $#pictographic_pairs; + print " ", join(', ', map { sprintf '0x%X', $pictographic_pairs[$_] } $i .. $end), ",\n"; +} +print <<'FOOTER'; + }; + + static byte propertyOf(int codePoint) { + int low = 0; + int high = STARTS.length - 1; + while (low <= high) { + int middle = (low + high) >>> 1; + if (STARTS[middle] <= codePoint) { + low = middle + 1; + } else { + high = middle - 1; + } + } + return VALUES[Math.max(0, high)]; + } + + static boolean isExtendedPictographic(int codePoint) { + int low = 0; + int high = EXTENDED_PICTOGRAPHIC_RANGES.length / 2 - 1; + while (low <= high) { + int middle = (low + high) >>> 1; + int start = EXTENDED_PICTOGRAPHIC_RANGES[middle * 2]; + int rangeEnd = EXTENDED_PICTOGRAPHIC_RANGES[middle * 2 + 1]; + if (codePoint < start) { + high = middle - 1; + } else if (codePoint > rangeEnd) { + low = middle + 1; + } else { + return true; + } + } + return false; + } + + private WordBreakData() { + } +} +FOOTER diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index d0c3f10f94..3b6df3c40d 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -385,7 +385,7 @@ my @copy = @{$z}; # ERROR - ✅ **Preprocessor**: `\Q`, `\L`, `\U`, `\l`, `\u`, `\E` are preprocessed in regex. - ✅ **Overloading**: `qr` overloading is implemented. See also [overload pragma](#pragmas). - ✅ **Python-style named groups**: `(?P...)` and `(?P=name)` are parsed natively by Joni with Perl capture numbering, duplicate-name behavior, and malformed/unknown-name diagnostics. -- ❌ **Alpha assertion aliases**: `(*pla:...)`, `(*plb:...)`, `(*nla:...)`, `(*nlb:...)`, and `(*atomic:...)` are not yet parsed by Joni. +- ✅ **Alpha assertion aliases**: `(*pla:...)`, `(*plb:...)`, `(*nla:...)`, `(*nlb:...)`, and `(*atomic:...)` are parsed natively by Joni with Perl nesting, capture numbering, backtracking, and malformed-form diagnostics. - ❌ **Underscored numeric regex escapes**: Perl spellings such as `\x{0_0_4_1}` and `\o{0_0_1_0_1}` still rely on frontend normalization rather than native Joni parsing. Valid braced octal and unconditional missing-close/empty diagnostics are native; three `use re 'strict'` non-octal cases remain deferred. - ✅ **Dynamically-scoped regex variables**: Provisional captures, `$^R`, `$^N`, match positions, and callback locals follow matcher paths and unwind on backtracking. @@ -397,9 +397,10 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 113,068/407,367 identically on JVM and interpreter: chunks 01–04 expose property/value alias gaps, and the remaining boundary failures are line and word breaks. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 132,578/407,367 identically on JVM and interpreter: chunks 01–04 expose property/value alias gaps, and the remaining boundary failures are line breaks. - ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. - ✅ **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter. +- ✅ **Unicode Word Boundaries**: Native `\b{wb}`/`\B{wb}` implement WB1–WB16 and WB999 from reproducible Perl 5.44 Unicode 17.0 word-break and extended-pictographic tables. Authoritative chunk 10 passes 19,510/19,510 identically on JVM and interpreter. - ✅ **Embedded Code in Regex**: `(?{ code })`, optimistic callbacks `(*{ code })`, executable callback conditions, and `(??{ code })` run as lexical closures in Joni with provisional captures and backtracking unwind. Callback `local` frames follow matcher paths, and escaped loop control or `goto` stops at the callback pseudo-block boundary. `$^N` follows capture-close order independently of `$+`. - ✅ **Regex Debugging**: Lexically scoped `use/no re 'debug'` and `debugcolor` are supported, including runtime snapshot ownership. - ✅ **Runtime Regex Evaluation**: `use re 'eval'` controls whether interpolated patterns containing eval groups may compile. Admitted runtime source is compiled into lexical callback closures and preserves its package, visible lexical cells, Unicode or byte source type, default regex modifiers, and match-once state. Literal callbacks, interpolated `qr//` values (including local, referenced, and tied arrays), raw runtime eval groups, callback conditions, and standalone `(?(DEFINE)...)` containers may be composed in one Joni pattern; dynamic callbacks may return further admitted executable source. diff --git a/src/test/resources/unit/regex/word_boundary_rules.t b/src/test/resources/unit/regex/word_boundary_rules.t new file mode 100644 index 0000000000..0c17f20f3e --- /dev/null +++ b/src/test/resources/unit/regex/word_boundary_rules.t @@ -0,0 +1,57 @@ +use strict; +use warnings; +use utf8; +use Test::More tests => 33; + +sub boundary_at { + my ($subject, $offset) = @_; + my $left = quotemeta substr($subject, 0, $offset); + my $right = quotemeta substr($subject, $offset); + return $subject =~ /\A${left}\b{wb}${right}\z/; +} + +sub no_boundary_at { + my ($subject, $offset) = @_; + my $left = quotemeta substr($subject, 0, $offset); + my $right = quotemeta substr($subject, $offset); + return $subject =~ /\A${left}\B{wb}${right}\z/; +} + +ok(boundary_at('ab', 0), 'WB1 start of text is a boundary'); +ok(boundary_at('ab', 2), 'WB2 end of text is a boundary'); +ok(no_boundary_at("\r\n", 1), 'WB3 keeps CR LF together'); +ok(boundary_at("\rA", 1), 'WB3a breaks after CR'); +ok(boundary_at("A\n", 1), 'WB3b breaks before LF'); +ok(no_boundary_at("\x{200D}\x{1F680}", 1), 'WB3c keeps ZWJ with pictograph'); +ok(boundary_at("\x{200D}\x{0308}\x{1F680}", 2), + 'WB3c does not skip Extend between ZWJ and pictograph'); +ok(no_boundary_at(' ', 1), 'WB3d keeps horizontal spaces together'); +ok(boundary_at(" \x{0308} ", 2), 'WB3d requires adjacent horizontal spaces'); +ok(no_boundary_at("A\x{0308}", 1), 'WB4 ignores Extend'); +ok(no_boundary_at('ab', 1), 'WB5 keeps letters together'); +ok(no_boundary_at('a.b', 1), 'WB6 keeps MidNumLet after a letter'); +ok(no_boundary_at('a.b', 2), 'WB7 keeps MidNumLet before a letter'); +ok(no_boundary_at("\x{05D0}'", 1), 'WB7a keeps Hebrew before single quote'); +ok(no_boundary_at("\x{05D0}\"\x{05D1}", 1), 'WB7b keeps Hebrew before double quote'); +ok(no_boundary_at("\x{05D0}\"\x{05D1}", 2), 'WB7c keeps Hebrew after double quote'); +ok(no_boundary_at('12', 1), 'WB8 keeps numeric characters together'); +ok(no_boundary_at('1,2', 1), 'WB12 keeps numeric punctuation after a number'); +ok(no_boundary_at('1,2', 2), 'WB11 keeps numeric punctuation before a number'); +ok(no_boundary_at('a1', 1), 'WB9 keeps letters before numeric characters'); +ok(no_boundary_at('1a', 1), 'WB10 keeps numeric characters before letters'); +ok(no_boundary_at("\x{30A2}\x{30A4}", 1), 'WB13 keeps Katakana together'); +ok(no_boundary_at('a_', 1), 'WB13a keeps ExtendNumLet after a letter'); +ok(no_boundary_at('_a', 1), 'WB13b keeps ExtendNumLet before a letter'); +ok(no_boundary_at("\x{1F1E6}\x{1F1E7}", 1), 'WB15 keeps the first RI pair'); +ok(boundary_at("\x{1F1E6}\x{1F1E7}\x{1F1E8}", 2), 'WB16 breaks before a third RI'); +ok(boundary_at('a,b', 1), 'WB999 breaks before unrelated punctuation'); +ok(boundary_at('a,b', 2), 'WB999 breaks after unrelated punctuation'); +ok(!boundary_at('ab', 1), 'positive assertion rejects a non-boundary'); +ok(!no_boundary_at('a,b', 1), 'negated assertion rejects a boundary'); +ok(no_boundary_at("a\x{00AD}b", 1), 'WB4 ignores Format before it'); +ok(no_boundary_at("a\x{00AD}b", 2), 'Format is transparent to letter context'); + +{ + use bytes; + ok('ab' =~ /\Aa\B{wb}b\z/, 'byte-mode ASCII uses word boundaries'); +} diff --git a/third_party/joni/src/org/joni/ArrayCompiler.java b/third_party/joni/src/org/joni/ArrayCompiler.java index fd66c8e15f..d3579bf3e4 100644 --- a/third_party/joni/src/org/joni/ArrayCompiler.java +++ b/third_party/joni/src/org/joni/ArrayCompiler.java @@ -1214,6 +1214,14 @@ protected void compileAnchorNode(AnchorNode node) { addOpcode(OPCode.NOT_SENTENCE_BOUNDARY); break; + case AnchorType.WORD_BREAK_BOUNDARY: + addOpcode(OPCode.WORD_BREAK_BOUNDARY); + break; + + case AnchorType.NOT_WORD_BREAK_BOUNDARY: + addOpcode(OPCode.NOT_WORD_BREAK_BOUNDARY); + break; + case AnchorType.WORD_BEGIN: if (Config.USE_WORD_BEGIN_END) { if (node.asciiRange) { diff --git a/third_party/joni/src/org/joni/ByteCodeMachine.java b/third_party/joni/src/org/joni/ByteCodeMachine.java index 36897666c6..c918696193 100644 --- a/third_party/joni/src/org/joni/ByteCodeMachine.java +++ b/third_party/joni/src/org/joni/ByteCodeMachine.java @@ -254,6 +254,8 @@ private final int execute(final boolean checkThreadInterrupt) throws Interrupted case OPCode.NOT_GRAPHEME_BOUNDARY: opGraphemeBoundary(true); continue; case OPCode.SENTENCE_BOUNDARY: opSentenceBoundary(false); continue; case OPCode.NOT_SENTENCE_BOUNDARY: opSentenceBoundary(true); continue; + case OPCode.WORD_BREAK_BOUNDARY: opWordBreakBoundary(false); continue; + case OPCode.NOT_WORD_BREAK_BOUNDARY: opWordBreakBoundary(true); continue; case OPCode.WORD_BEGIN: opWordBegin(); continue; case OPCode.WORD_END: opWordEnd(); continue; @@ -415,6 +417,8 @@ private final int executeSb(final boolean checkThreadInterrupt) throws Interrupt case OPCode.NOT_GRAPHEME_BOUNDARY: opGraphemeBoundary(true); continue; case OPCode.SENTENCE_BOUNDARY: opSentenceBoundary(false); continue; case OPCode.NOT_SENTENCE_BOUNDARY: opSentenceBoundary(true); continue; + case OPCode.WORD_BREAK_BOUNDARY: opWordBreakBoundary(false); continue; + case OPCode.NOT_WORD_BREAK_BOUNDARY: opWordBreakBoundary(true); continue; case OPCode.WORD_BEGIN: opWordBeginSb(); continue; case OPCode.WORD_END: opWordEndSb(); continue; @@ -1255,6 +1259,134 @@ private void opSentenceBoundary(boolean negated) { if (isSentenceBoundary() == negated) opFail(); } + private void opWordBreakBoundary(boolean negated) { + if (isWordBreakBoundary() == negated) opFail(); + } + + private boolean isWordBreakBoundary() { + if (s <= str || s >= end) return true; // WB1, WB2 + + int leftPosition = enc.prevCharHead(bytes, str, s, end); + byte immediateLeft = wordPropertyAt(leftPosition); + byte right = wordPropertyAt(s); + + if (immediateLeft == WordBreakData.CR && right == WordBreakData.LF) return false; // WB3 + if (isWordNewline(immediateLeft) || isWordNewline(right)) return true; // WB3a, WB3b + if (isWordIgnored(right)) return false; // WB4 + + if (immediateLeft == WordBreakData.ZWJ + && WordBreakData.isExtendedPictographic(enc.mbcToCode(bytes, s, end))) return false; // WB3c + + int[] leftCursor = {leftPosition}; + byte left = previousWordProperty(leftCursor); + + if (immediateLeft == WordBreakData.WSEG_SPACE + && right == WordBreakData.WSEG_SPACE) return false; // WB3d + + if (isWordAHLetter(left) && isWordAHLetter(right)) return false; // WB5 + if (isWordAHLetter(left) && isWordLetterMid(right) + && isWordAHLetter(nextWordPropertyAfter(s))) return false; // WB6 + if (isWordLetterMid(left) && isWordAHLetter(right) + && isWordAHLetter(previousWordProperty(leftCursor))) return false; // WB7 + + if (left == WordBreakData.HEBREW_LETTER + && right == WordBreakData.SINGLE_QUOTE) return false; // WB7a + if (left == WordBreakData.HEBREW_LETTER + && right == WordBreakData.DOUBLE_QUOTE + && nextWordPropertyAfter(s) == WordBreakData.HEBREW_LETTER) return false; // WB7b + if (left == WordBreakData.DOUBLE_QUOTE + && right == WordBreakData.HEBREW_LETTER + && previousWordProperty(leftCursor) == WordBreakData.HEBREW_LETTER) return false; // WB7c + + if (left == WordBreakData.NUMERIC && right == WordBreakData.NUMERIC) return false; // WB8 + if (isWordAHLetter(left) && right == WordBreakData.NUMERIC) return false; // WB9 + if (left == WordBreakData.NUMERIC && isWordAHLetter(right)) return false; // WB10 + if (isWordNumericMid(left) && right == WordBreakData.NUMERIC + && previousWordProperty(leftCursor) == WordBreakData.NUMERIC) return false; // WB11 + if (left == WordBreakData.NUMERIC && isWordNumericMid(right) + && nextWordPropertyAfter(s) == WordBreakData.NUMERIC) return false; // WB12 + + if (left == WordBreakData.KATAKANA && right == WordBreakData.KATAKANA) return false; // WB13 + if (isWordExtendNumLetBase(left) && right == WordBreakData.EXTEND_NUM_LET) return false; // WB13a + if (left == WordBreakData.EXTEND_NUM_LET && isWordExtendNumLetBase(right)) return false; // WB13b + + if (left == WordBreakData.REGIONAL_INDICATOR + && right == WordBreakData.REGIONAL_INDICATOR + && hasOddWordRegionalIndicatorRun(leftPosition)) return false; // WB15, WB16 + + return true; // WB999 + } + + private byte wordPropertyAt(int position) { + return WordBreakData.propertyOf(enc.mbcToCode(bytes, position, end)); + } + + private byte previousWordProperty(int[] position) { + while (position[0] >= str) { + byte property = wordPropertyAt(position[0]); + if (position[0] == str) { + position[0] = -1; + } else { + position[0] = enc.prevCharHead(bytes, str, position[0], end); + } + if (!isWordIgnored(property)) return property; + } + return WordBreakData.OTHER; + } + + private byte nextWordPropertyAfter(int position) { + position += enc.length(bytes, position, end); + while (position < end) { + byte property = wordPropertyAt(position); + if (!isWordIgnored(property)) return property; + position += enc.length(bytes, position, end); + } + return WordBreakData.OTHER; + } + + private boolean hasOddWordRegionalIndicatorRun(int leftPosition) { + int count = 0; + int[] position = {leftPosition}; + while (previousWordProperty(position) == WordBreakData.REGIONAL_INDICATOR) count++; + return (count & 1) == 1; + } + + private boolean isWordIgnored(byte property) { + return property == WordBreakData.EXTEND + || property == WordBreakData.FORMAT + || property == WordBreakData.ZWJ; + } + + private boolean isWordNewline(byte property) { + return property == WordBreakData.NEWLINE + || property == WordBreakData.CR + || property == WordBreakData.LF; + } + + private boolean isWordAHLetter(byte property) { + return property == WordBreakData.ALETTER + || property == WordBreakData.HEBREW_LETTER; + } + + private boolean isWordLetterMid(byte property) { + return property == WordBreakData.MID_LETTER + || property == WordBreakData.MID_NUM_LET + || property == WordBreakData.SINGLE_QUOTE; + } + + private boolean isWordNumericMid(byte property) { + return property == WordBreakData.MID_NUM + || property == WordBreakData.MID_NUM_LET + || property == WordBreakData.SINGLE_QUOTE; + } + + private boolean isWordExtendNumLetBase(byte property) { + return isWordAHLetter(property) + || property == WordBreakData.NUMERIC + || property == WordBreakData.KATAKANA + || property == WordBreakData.EXTEND_NUM_LET; + } + private boolean isSentenceBoundary() { if (s <= str || s >= end) return true; // SB1, SB2 diff --git a/third_party/joni/src/org/joni/Lexer.java b/third_party/joni/src/org/joni/Lexer.java index b32d8413ef..109acc1fbf 100644 --- a/third_party/joni/src/org/joni/Lexer.java +++ b/third_party/joni/src/org/joni/Lexer.java @@ -1448,6 +1448,12 @@ private boolean fetchTokenForPerlBoundary(boolean negated) { : AnchorType.SENTENCE_BOUNDARY); return true; } + if (name.toString().equals("wb")) { + fetchTokenFor_anchor(negated + ? AnchorType.NOT_WORD_BREAK_BOUNDARY + : AnchorType.WORD_BREAK_BOUNDARY); + return true; + } restore(); return false; } diff --git a/third_party/joni/src/org/joni/Parser.java b/third_party/joni/src/org/joni/Parser.java index 5eeac74765..247be8edce 100644 --- a/third_party/joni/src/org/joni/Parser.java +++ b/third_party/joni/src/org/joni/Parser.java @@ -893,6 +893,9 @@ private Node parseInternalCallout() { } private Node parseControlVerb() { + Node alphaAssertion = parseAlphaAssertion(); + if (alphaAssertion != null) return alphaAssertion; + final ControlVerbNode.Kind kind; final String verb; if (startsWith("ACCEPT)")) { @@ -920,7 +923,11 @@ private Node parseControlVerb() { kind = ControlVerbNode.Kind.MARK; verb = "MARK"; } else { - newSyntaxException(UNDEFINED_GROUP_OPTION); + String construct = controlConstructName(); + if (!construct.isEmpty() && Character.isUpperCase(construct.codePointAt(0))) { + newValueException(PERL_UNKNOWN_VERB_PATTERN, construct); + } + newValueException(PERL_UNKNOWN_CONTROL_CONSTRUCT, construct); return null; } p += verb.length(); @@ -939,6 +946,73 @@ private Node parseControlVerb() { return new ControlVerbNode(kind, name); } + private Node parseAlphaAssertion() { + int nameStart = p; + int cursor = p; + while (cursor < stop) { + int code = enc.mbcToCode(bytes, cursor, stop); + if (!Character.isLetter(code)) break; + cursor += enc.length(bytes, cursor, stop); + } + String name = new String(bytes, nameStart, cursor - nameStart, StandardCharsets.UTF_8); + if (!name.equals("pla") && !name.equals("plb") && !name.equals("nla") + && !name.equals("nlb") && !name.equals("atomic")) { + return null; + } + if (cursor >= stop || enc.mbcToCode(bytes, cursor, stop) != ':') { + newValueException(PERL_ALPHA_ASSERTION_REQUIRES_COLON, name); + } + p = cursor + enc.length(bytes, cursor, stop); + + Node node; + switch (name) { + case "pla": + node = new AnchorNode(AnchorType.PREC_READ); + break; + case "plb": + node = new AnchorNode(AnchorType.LOOK_BEHIND); + break; + case "nla": + node = new AnchorNode(AnchorType.PREC_READ_NOT); + break; + case "nlb": + node = new AnchorNode(AnchorType.LOOK_BEHIND_NOT); + break; + default: + node = new EncloseNode(EncloseType.STOP_BACKTRACK); + break; + } + + fetchToken(); + final Node target; + try { + target = parseSubExp(TokenType.SUBEXP_CLOSE); + } catch (SyntaxException e) { + if (END_PATTERN_WITH_UNMATCHED_PARENTHESIS.equals(e.getMessage())) { + newSyntaxException(PERL_UNTERMINATED_CONTROL_ARGUMENT); + } + throw e; + } + if (node.getType() == NodeType.ANCHOR) { + ((AnchorNode)node).setTarget(target); + } else { + ((EncloseNode)node).setTarget(target); + } + returnCode = 0; + return node; + } + + private String controlConstructName() { + int start = p; + int cursor = p; + while (cursor < stop) { + int code = enc.mbcToCode(bytes, cursor, stop); + if (code == ':' || code == ')') break; + cursor += enc.length(bytes, cursor, stop); + } + return new String(bytes, start, cursor - start, StandardCharsets.UTF_8); + } + private int parseInternalCalloutId() { return parseInternalCalloutId("=CALL:"); } diff --git a/third_party/joni/src/org/joni/WordBreakData.java b/third_party/joni/src/org/joni/WordBreakData.java new file mode 100644 index 0000000000..432668da2f --- /dev/null +++ b/third_party/joni/src/org/joni/WordBreakData.java @@ -0,0 +1,567 @@ +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni; + +// Generated by dev/tools/generate_joni_word_break_data.pl. Do not edit manually. +final class WordBreakData { + static final String UNICODE_VERSION = "17.0.0"; + static final byte OTHER = 0; + static final byte ALETTER = 1; + static final byte CR = 2; + static final byte DOUBLE_QUOTE = 3; + static final byte EXTEND = 4; + static final byte EXTEND_NUM_LET = 5; + static final byte FORMAT = 6; + static final byte HEBREW_LETTER = 7; + static final byte KATAKANA = 8; + static final byte LF = 9; + static final byte MID_LETTER = 10; + static final byte MID_NUM = 11; + static final byte MID_NUM_LET = 12; + static final byte NEWLINE = 13; + static final byte NUMERIC = 14; + static final byte REGIONAL_INDICATOR = 15; + static final byte SINGLE_QUOTE = 16; + static final byte WSEG_SPACE = 17; + static final byte ZWJ = 18; + + private static final int[] STARTS = { + 0x0, 0xA, 0xB, 0xD, 0xE, 0x20, 0x21, 0x22, 0x23, 0x27, + 0x28, 0x2C, 0x2D, 0x2E, 0x2F, 0x30, 0x3A, 0x3B, 0x3C, 0x41, + 0x5B, 0x5F, 0x60, 0x61, 0x7B, 0x85, 0x86, 0xAA, 0xAB, 0xAD, + 0xAE, 0xB5, 0xB6, 0xB7, 0xB8, 0xB9, 0xBA, 0xBB, 0xC0, 0xD7, + 0xD8, 0xF7, 0xF8, 0x1BB, 0x1BC, 0x1C0, 0x1C4, 0x294, 0x296, 0x2B0, + 0x2C2, 0x2C6, 0x2D2, 0x2D8, 0x2DE, 0x2E0, 0x2E5, 0x2EC, 0x2ED, 0x2EE, + 0x2EF, 0x300, 0x370, 0x374, 0x375, 0x376, 0x378, 0x37A, 0x37B, 0x37E, + 0x37F, 0x380, 0x386, 0x387, 0x388, 0x38B, 0x38C, 0x38D, 0x38E, 0x3A2, + 0x3A3, 0x3F6, 0x3F7, 0x482, 0x483, 0x488, 0x48A, 0x530, 0x531, 0x557, + 0x559, 0x55A, 0x55D, 0x55E, 0x55F, 0x560, 0x589, 0x58A, 0x58B, 0x591, + 0x5BE, 0x5BF, 0x5C0, 0x5C1, 0x5C3, 0x5C4, 0x5C6, 0x5C7, 0x5C8, 0x5D0, + 0x5EB, 0x5EF, 0x5F3, 0x5F4, 0x5F5, 0x600, 0x606, 0x60C, 0x60E, 0x610, + 0x61B, 0x61C, 0x61D, 0x620, 0x640, 0x641, 0x64B, 0x660, 0x66A, 0x66B, + 0x66C, 0x66D, 0x66E, 0x670, 0x671, 0x6D4, 0x6D5, 0x6D6, 0x6DD, 0x6DE, + 0x6DF, 0x6E5, 0x6E7, 0x6E9, 0x6EA, 0x6EE, 0x6F0, 0x6FA, 0x6FD, 0x6FF, + 0x700, 0x70F, 0x710, 0x711, 0x712, 0x730, 0x74B, 0x74D, 0x7A6, 0x7B1, + 0x7B2, 0x7C0, 0x7CA, 0x7EB, 0x7F4, 0x7F6, 0x7F8, 0x7F9, 0x7FA, 0x7FB, + 0x7FD, 0x7FE, 0x800, 0x816, 0x81A, 0x81B, 0x824, 0x825, 0x828, 0x829, + 0x82E, 0x840, 0x859, 0x85C, 0x860, 0x86B, 0x870, 0x888, 0x889, 0x890, + 0x892, 0x897, 0x8A0, 0x8C9, 0x8CA, 0x8E2, 0x8E3, 0x903, 0x904, 0x93A, + 0x93B, 0x93C, 0x93D, 0x93E, 0x941, 0x949, 0x94D, 0x94E, 0x950, 0x951, + 0x958, 0x962, 0x964, 0x966, 0x970, 0x971, 0x972, 0x981, 0x982, 0x984, + 0x985, 0x98D, 0x98F, 0x991, 0x993, 0x9A9, 0x9AA, 0x9B1, 0x9B2, 0x9B3, + 0x9B6, 0x9BA, 0x9BC, 0x9BD, 0x9BE, 0x9C1, 0x9C5, 0x9C7, 0x9C9, 0x9CB, + 0x9CD, 0x9CE, 0x9CF, 0x9D7, 0x9D8, 0x9DC, 0x9DE, 0x9DF, 0x9E2, 0x9E4, + 0x9E6, 0x9F0, 0x9F2, 0x9FC, 0x9FD, 0x9FE, 0x9FF, 0xA01, 0xA03, 0xA04, + 0xA05, 0xA0B, 0xA0F, 0xA11, 0xA13, 0xA29, 0xA2A, 0xA31, 0xA32, 0xA34, + 0xA35, 0xA37, 0xA38, 0xA3A, 0xA3C, 0xA3D, 0xA3E, 0xA41, 0xA43, 0xA47, + 0xA49, 0xA4B, 0xA4E, 0xA51, 0xA52, 0xA59, 0xA5D, 0xA5E, 0xA5F, 0xA66, + 0xA70, 0xA72, 0xA75, 0xA76, 0xA81, 0xA83, 0xA84, 0xA85, 0xA8E, 0xA8F, + 0xA92, 0xA93, 0xAA9, 0xAAA, 0xAB1, 0xAB2, 0xAB4, 0xAB5, 0xABA, 0xABC, + 0xABD, 0xABE, 0xAC1, 0xAC6, 0xAC7, 0xAC9, 0xACA, 0xACB, 0xACD, 0xACE, + 0xAD0, 0xAD1, 0xAE0, 0xAE2, 0xAE4, 0xAE6, 0xAF0, 0xAF9, 0xAFA, 0xB00, + 0xB01, 0xB02, 0xB04, 0xB05, 0xB0D, 0xB0F, 0xB11, 0xB13, 0xB29, 0xB2A, + 0xB31, 0xB32, 0xB34, 0xB35, 0xB3A, 0xB3C, 0xB3D, 0xB3E, 0xB3F, 0xB40, + 0xB41, 0xB45, 0xB47, 0xB49, 0xB4B, 0xB4D, 0xB4E, 0xB55, 0xB57, 0xB58, + 0xB5C, 0xB5E, 0xB5F, 0xB62, 0xB64, 0xB66, 0xB70, 0xB71, 0xB72, 0xB82, + 0xB83, 0xB84, 0xB85, 0xB8B, 0xB8E, 0xB91, 0xB92, 0xB96, 0xB99, 0xB9B, + 0xB9C, 0xB9D, 0xB9E, 0xBA0, 0xBA3, 0xBA5, 0xBA8, 0xBAB, 0xBAE, 0xBBA, + 0xBBE, 0xBC0, 0xBC1, 0xBC3, 0xBC6, 0xBC9, 0xBCA, 0xBCD, 0xBCE, 0xBD0, + 0xBD1, 0xBD7, 0xBD8, 0xBE6, 0xBF0, 0xC00, 0xC01, 0xC04, 0xC05, 0xC0D, + 0xC0E, 0xC11, 0xC12, 0xC29, 0xC2A, 0xC3A, 0xC3C, 0xC3D, 0xC3E, 0xC41, + 0xC45, 0xC46, 0xC49, 0xC4A, 0xC4E, 0xC55, 0xC57, 0xC58, 0xC5B, 0xC5C, + 0xC5E, 0xC60, 0xC62, 0xC64, 0xC66, 0xC70, 0xC80, 0xC81, 0xC82, 0xC84, + 0xC85, 0xC8D, 0xC8E, 0xC91, 0xC92, 0xCA9, 0xCAA, 0xCB4, 0xCB5, 0xCBA, + 0xCBC, 0xCBD, 0xCBE, 0xCBF, 0xCC0, 0xCC5, 0xCC6, 0xCC7, 0xCC9, 0xCCA, + 0xCCC, 0xCCE, 0xCD5, 0xCD7, 0xCDC, 0xCDF, 0xCE0, 0xCE2, 0xCE4, 0xCE6, + 0xCF0, 0xCF1, 0xCF3, 0xCF4, 0xD00, 0xD02, 0xD04, 0xD0D, 0xD0E, 0xD11, + 0xD12, 0xD3B, 0xD3D, 0xD3E, 0xD41, 0xD45, 0xD46, 0xD49, 0xD4A, 0xD4D, + 0xD4E, 0xD4F, 0xD54, 0xD57, 0xD58, 0xD5F, 0xD62, 0xD64, 0xD66, 0xD70, + 0xD7A, 0xD80, 0xD81, 0xD82, 0xD84, 0xD85, 0xD97, 0xD9A, 0xDB2, 0xDB3, + 0xDBC, 0xDBD, 0xDBE, 0xDC0, 0xDC7, 0xDCA, 0xDCB, 0xDCF, 0xDD2, 0xDD5, + 0xDD6, 0xDD7, 0xDD8, 0xDE0, 0xDE6, 0xDF0, 0xDF2, 0xDF4, 0xE31, 0xE32, + 0xE34, 0xE3B, 0xE47, 0xE4F, 0xE50, 0xE5A, 0xEB1, 0xEB2, 0xEB4, 0xEBD, + 0xEC8, 0xECF, 0xED0, 0xEDA, 0xF00, 0xF01, 0xF18, 0xF1A, 0xF20, 0xF2A, + 0xF35, 0xF36, 0xF37, 0xF38, 0xF39, 0xF3A, 0xF3E, 0xF40, 0xF48, 0xF49, + 0xF6D, 0xF71, 0xF7F, 0xF80, 0xF85, 0xF86, 0xF88, 0xF8D, 0xF98, 0xF99, + 0xFBD, 0xFC6, 0xFC7, 0x102B, 0x102D, 0x1031, 0x1032, 0x1038, 0x1039, 0x103B, + 0x103D, 0x103F, 0x1040, 0x104A, 0x1056, 0x1058, 0x105A, 0x105E, 0x1061, 0x1062, + 0x1065, 0x1067, 0x106E, 0x1071, 0x1075, 0x1082, 0x1083, 0x1085, 0x1087, 0x108D, + 0x108E, 0x108F, 0x1090, 0x109A, 0x109D, 0x109E, 0x10A0, 0x10C6, 0x10C7, 0x10C8, + 0x10CD, 0x10CE, 0x10D0, 0x10FB, 0x10FC, 0x10FD, 0x1100, 0x1249, 0x124A, 0x124E, + 0x1250, 0x1257, 0x1258, 0x1259, 0x125A, 0x125E, 0x1260, 0x1289, 0x128A, 0x128E, + 0x1290, 0x12B1, 0x12B2, 0x12B6, 0x12B8, 0x12BF, 0x12C0, 0x12C1, 0x12C2, 0x12C6, + 0x12C8, 0x12D7, 0x12D8, 0x1311, 0x1312, 0x1316, 0x1318, 0x135B, 0x135D, 0x1360, + 0x1380, 0x1390, 0x13A0, 0x13F6, 0x13F8, 0x13FE, 0x1401, 0x166D, 0x166F, 0x1680, + 0x1681, 0x169B, 0x16A0, 0x16EB, 0x16EE, 0x16F1, 0x16F9, 0x1700, 0x1712, 0x1715, + 0x1716, 0x171F, 0x1732, 0x1734, 0x1735, 0x1740, 0x1752, 0x1754, 0x1760, 0x176D, + 0x176E, 0x1771, 0x1772, 0x1774, 0x17B4, 0x17B6, 0x17B7, 0x17BE, 0x17C6, 0x17C7, + 0x17C9, 0x17D4, 0x17DD, 0x17DE, 0x17E0, 0x17EA, 0x180B, 0x180E, 0x180F, 0x1810, + 0x181A, 0x1820, 0x1843, 0x1844, 0x1879, 0x1880, 0x1885, 0x1887, 0x18A9, 0x18AA, + 0x18AB, 0x18B0, 0x18F6, 0x1900, 0x191F, 0x1920, 0x1923, 0x1927, 0x1929, 0x192C, + 0x1930, 0x1932, 0x1933, 0x1939, 0x193C, 0x1946, 0x1950, 0x19D0, 0x19DA, 0x19DB, + 0x1A00, 0x1A17, 0x1A19, 0x1A1B, 0x1A1C, 0x1A55, 0x1A56, 0x1A57, 0x1A58, 0x1A5F, + 0x1A60, 0x1A61, 0x1A62, 0x1A63, 0x1A65, 0x1A6D, 0x1A73, 0x1A7D, 0x1A7F, 0x1A80, + 0x1A8A, 0x1A90, 0x1A9A, 0x1AB0, 0x1ABE, 0x1ABF, 0x1ADE, 0x1AE0, 0x1AEC, 0x1B00, + 0x1B04, 0x1B05, 0x1B34, 0x1B35, 0x1B36, 0x1B3B, 0x1B3C, 0x1B3D, 0x1B42, 0x1B43, + 0x1B45, 0x1B4D, 0x1B50, 0x1B5A, 0x1B6B, 0x1B74, 0x1B80, 0x1B82, 0x1B83, 0x1BA1, + 0x1BA2, 0x1BA6, 0x1BA8, 0x1BAA, 0x1BAB, 0x1BAE, 0x1BB0, 0x1BBA, 0x1BE6, 0x1BE7, + 0x1BE8, 0x1BEA, 0x1BED, 0x1BEE, 0x1BEF, 0x1BF2, 0x1BF4, 0x1C00, 0x1C24, 0x1C2C, + 0x1C34, 0x1C36, 0x1C38, 0x1C40, 0x1C4A, 0x1C4D, 0x1C50, 0x1C5A, 0x1C78, 0x1C7E, + 0x1C80, 0x1C8B, 0x1C90, 0x1CBB, 0x1CBD, 0x1CC0, 0x1CD0, 0x1CD3, 0x1CD4, 0x1CE1, + 0x1CE2, 0x1CE9, 0x1CED, 0x1CEE, 0x1CF4, 0x1CF5, 0x1CF7, 0x1CF8, 0x1CFA, 0x1CFB, + 0x1D00, 0x1D2C, 0x1D6B, 0x1D78, 0x1D79, 0x1D9B, 0x1DC0, 0x1E00, 0x1F16, 0x1F18, + 0x1F1E, 0x1F20, 0x1F46, 0x1F48, 0x1F4E, 0x1F50, 0x1F58, 0x1F59, 0x1F5A, 0x1F5B, + 0x1F5C, 0x1F5D, 0x1F5E, 0x1F5F, 0x1F7E, 0x1F80, 0x1FB5, 0x1FB6, 0x1FBD, 0x1FBE, + 0x1FBF, 0x1FC2, 0x1FC5, 0x1FC6, 0x1FCD, 0x1FD0, 0x1FD4, 0x1FD6, 0x1FDC, 0x1FE0, + 0x1FED, 0x1FF2, 0x1FF5, 0x1FF6, 0x1FFD, 0x2000, 0x2007, 0x2008, 0x200B, 0x200C, + 0x200D, 0x200E, 0x2010, 0x2018, 0x2019, 0x201A, 0x2024, 0x2025, 0x2027, 0x2028, + 0x2029, 0x202A, 0x202F, 0x2030, 0x203F, 0x2041, 0x2044, 0x2045, 0x2054, 0x2055, + 0x205F, 0x2060, 0x2065, 0x2066, 0x2070, 0x2071, 0x2072, 0x207F, 0x2080, 0x2090, + 0x209D, 0x20D0, 0x20DD, 0x20E1, 0x20E2, 0x20E5, 0x20F1, 0x2102, 0x2103, 0x2107, + 0x2108, 0x210A, 0x2114, 0x2115, 0x2116, 0x2119, 0x211E, 0x2124, 0x2125, 0x2126, + 0x2127, 0x2128, 0x2129, 0x212A, 0x212E, 0x212F, 0x2135, 0x2139, 0x213A, 0x213C, + 0x2140, 0x2145, 0x214A, 0x214E, 0x214F, 0x2160, 0x2183, 0x2185, 0x2189, 0x24B6, + 0x24EA, 0x2C00, 0x2C7C, 0x2C7E, 0x2CE5, 0x2CEB, 0x2CEF, 0x2CF2, 0x2CF4, 0x2D00, + 0x2D26, 0x2D27, 0x2D28, 0x2D2D, 0x2D2E, 0x2D30, 0x2D68, 0x2D6F, 0x2D70, 0x2D7F, + 0x2D80, 0x2D97, 0x2DA0, 0x2DA7, 0x2DA8, 0x2DAF, 0x2DB0, 0x2DB7, 0x2DB8, 0x2DBF, + 0x2DC0, 0x2DC7, 0x2DC8, 0x2DCF, 0x2DD0, 0x2DD7, 0x2DD8, 0x2DDF, 0x2DE0, 0x2E00, + 0x2E2F, 0x2E30, 0x3000, 0x3001, 0x3005, 0x3006, 0x302A, 0x302E, 0x3030, 0x3031, + 0x3036, 0x303B, 0x303C, 0x303D, 0x3099, 0x309B, 0x309D, 0x30A0, 0x30A1, 0x30FB, + 0x30FC, 0x30FF, 0x3100, 0x3105, 0x3130, 0x3131, 0x318F, 0x31A0, 0x31C0, 0x31F0, + 0x3200, 0x32D0, 0x32FF, 0x3300, 0x3358, 0xA000, 0xA015, 0xA016, 0xA48D, 0xA4D0, + 0xA4F8, 0xA4FE, 0xA500, 0xA60C, 0xA60D, 0xA610, 0xA620, 0xA62A, 0xA62C, 0xA640, + 0xA66E, 0xA66F, 0xA670, 0xA673, 0xA674, 0xA67E, 0xA67F, 0xA680, 0xA69C, 0xA69E, + 0xA6A0, 0xA6E6, 0xA6F0, 0xA6F2, 0xA708, 0xA717, 0xA720, 0xA722, 0xA770, 0xA771, + 0xA788, 0xA789, 0xA78B, 0xA78F, 0xA790, 0xA7DD, 0xA7F1, 0xA7F5, 0xA7F7, 0xA7F8, + 0xA7FA, 0xA7FB, 0xA802, 0xA803, 0xA806, 0xA807, 0xA80B, 0xA80C, 0xA823, 0xA825, + 0xA827, 0xA828, 0xA82C, 0xA82D, 0xA840, 0xA874, 0xA880, 0xA882, 0xA8B4, 0xA8C4, + 0xA8C6, 0xA8D0, 0xA8DA, 0xA8E0, 0xA8F2, 0xA8F8, 0xA8FB, 0xA8FC, 0xA8FD, 0xA8FF, + 0xA900, 0xA90A, 0xA926, 0xA92E, 0xA930, 0xA947, 0xA952, 0xA954, 0xA960, 0xA97D, + 0xA980, 0xA983, 0xA984, 0xA9B3, 0xA9B4, 0xA9B6, 0xA9BA, 0xA9BC, 0xA9BE, 0xA9C1, + 0xA9CF, 0xA9D0, 0xA9DA, 0xA9E5, 0xA9E6, 0xA9F0, 0xA9FA, 0xAA00, 0xAA29, 0xAA2F, + 0xAA31, 0xAA33, 0xAA35, 0xAA37, 0xAA40, 0xAA43, 0xAA44, 0xAA4C, 0xAA4D, 0xAA4E, + 0xAA50, 0xAA5A, 0xAA7B, 0xAA7C, 0xAA7D, 0xAA7E, 0xAAB0, 0xAAB1, 0xAAB2, 0xAAB5, + 0xAAB7, 0xAAB9, 0xAABE, 0xAAC0, 0xAAC1, 0xAAC2, 0xAAE0, 0xAAEB, 0xAAEC, 0xAAEE, + 0xAAF0, 0xAAF2, 0xAAF3, 0xAAF5, 0xAAF6, 0xAAF7, 0xAB01, 0xAB07, 0xAB09, 0xAB0F, + 0xAB11, 0xAB17, 0xAB20, 0xAB27, 0xAB28, 0xAB2F, 0xAB30, 0xAB5B, 0xAB5C, 0xAB60, + 0xAB69, 0xAB6A, 0xAB70, 0xABC0, 0xABE3, 0xABE5, 0xABE6, 0xABE8, 0xABE9, 0xABEB, + 0xABEC, 0xABED, 0xABEE, 0xABF0, 0xABFA, 0xAC00, 0xD7A4, 0xD7B0, 0xD7C7, 0xD7CB, + 0xD7FC, 0xFB00, 0xFB07, 0xFB13, 0xFB18, 0xFB1D, 0xFB1E, 0xFB1F, 0xFB29, 0xFB2A, + 0xFB37, 0xFB38, 0xFB3D, 0xFB3E, 0xFB3F, 0xFB40, 0xFB42, 0xFB43, 0xFB45, 0xFB46, + 0xFB50, 0xFBB2, 0xFBD3, 0xFD3E, 0xFD50, 0xFD90, 0xFD92, 0xFDC8, 0xFDF0, 0xFDFC, + 0xFE00, 0xFE10, 0xFE13, 0xFE14, 0xFE20, 0xFE30, 0xFE33, 0xFE35, 0xFE4D, 0xFE50, + 0xFE51, 0xFE52, 0xFE53, 0xFE54, 0xFE55, 0xFE56, 0xFE70, 0xFE75, 0xFE76, 0xFEFD, + 0xFEFF, 0xFF00, 0xFF07, 0xFF08, 0xFF0C, 0xFF0D, 0xFF0E, 0xFF0F, 0xFF10, 0xFF1A, + 0xFF1B, 0xFF1C, 0xFF21, 0xFF3B, 0xFF3F, 0xFF40, 0xFF41, 0xFF5B, 0xFF66, 0xFF70, + 0xFF71, 0xFF9E, 0xFFA0, 0xFFBF, 0xFFC2, 0xFFC8, 0xFFCA, 0xFFD0, 0xFFD2, 0xFFD8, + 0xFFDA, 0xFFDD, 0xFFF9, 0xFFFC, 0x10000, 0x1000C, 0x1000D, 0x10027, 0x10028, 0x1003B, + 0x1003C, 0x1003E, 0x1003F, 0x1004E, 0x10050, 0x1005E, 0x10080, 0x100FB, 0x10140, 0x10175, + 0x101FD, 0x101FE, 0x10280, 0x1029D, 0x102A0, 0x102D1, 0x102E0, 0x102E1, 0x10300, 0x10320, + 0x1032D, 0x10341, 0x10342, 0x1034A, 0x1034B, 0x10350, 0x10376, 0x1037B, 0x10380, 0x1039E, + 0x103A0, 0x103C4, 0x103C8, 0x103D0, 0x103D1, 0x103D6, 0x10400, 0x10450, 0x1049E, 0x104A0, + 0x104AA, 0x104B0, 0x104D4, 0x104D8, 0x104FC, 0x10500, 0x10528, 0x10530, 0x10564, 0x10570, + 0x1057B, 0x1057C, 0x1058B, 0x1058C, 0x10593, 0x10594, 0x10596, 0x10597, 0x105A2, 0x105A3, + 0x105B2, 0x105B3, 0x105BA, 0x105BB, 0x105BD, 0x105C0, 0x105F4, 0x10600, 0x10737, 0x10740, + 0x10756, 0x10760, 0x10768, 0x10780, 0x10786, 0x10787, 0x107B1, 0x107B2, 0x107BB, 0x10800, + 0x10806, 0x10808, 0x10809, 0x1080A, 0x10836, 0x10837, 0x10839, 0x1083C, 0x1083D, 0x1083F, + 0x10856, 0x10860, 0x10877, 0x10880, 0x1089F, 0x108E0, 0x108F3, 0x108F4, 0x108F6, 0x10900, + 0x10916, 0x10920, 0x1093A, 0x10940, 0x1095A, 0x10980, 0x109B8, 0x109BE, 0x109C0, 0x10A00, + 0x10A01, 0x10A04, 0x10A05, 0x10A07, 0x10A0C, 0x10A10, 0x10A14, 0x10A15, 0x10A18, 0x10A19, + 0x10A36, 0x10A38, 0x10A3B, 0x10A3F, 0x10A40, 0x10A60, 0x10A7D, 0x10A80, 0x10A9D, 0x10AC0, + 0x10AC8, 0x10AC9, 0x10AE5, 0x10AE7, 0x10B00, 0x10B36, 0x10B40, 0x10B56, 0x10B60, 0x10B73, + 0x10B80, 0x10B92, 0x10C00, 0x10C49, 0x10C80, 0x10CB3, 0x10CC0, 0x10CF3, 0x10D00, 0x10D24, + 0x10D28, 0x10D30, 0x10D3A, 0x10D40, 0x10D4A, 0x10D4E, 0x10D4F, 0x10D50, 0x10D66, 0x10D69, + 0x10D6E, 0x10D6F, 0x10D70, 0x10D86, 0x10E80, 0x10EAA, 0x10EAB, 0x10EAD, 0x10EB0, 0x10EB2, + 0x10EC2, 0x10EC5, 0x10EC6, 0x10EC8, 0x10EFA, 0x10F00, 0x10F1D, 0x10F27, 0x10F28, 0x10F30, + 0x10F46, 0x10F51, 0x10F70, 0x10F82, 0x10F86, 0x10FB0, 0x10FC5, 0x10FE0, 0x10FF7, 0x11000, + 0x11001, 0x11002, 0x11003, 0x11038, 0x11047, 0x11066, 0x11070, 0x11071, 0x11073, 0x11075, + 0x11076, 0x1107F, 0x11082, 0x11083, 0x110B0, 0x110B3, 0x110B7, 0x110B9, 0x110BB, 0x110BD, + 0x110BE, 0x110C2, 0x110C3, 0x110CD, 0x110CE, 0x110D0, 0x110E9, 0x110F0, 0x110FA, 0x11100, + 0x11103, 0x11127, 0x1112C, 0x1112D, 0x11135, 0x11136, 0x11140, 0x11144, 0x11145, 0x11147, + 0x11148, 0x11150, 0x11173, 0x11174, 0x11176, 0x11177, 0x11180, 0x11182, 0x11183, 0x111B3, + 0x111B6, 0x111BF, 0x111C1, 0x111C5, 0x111C9, 0x111CD, 0x111CE, 0x111CF, 0x111D0, 0x111DA, + 0x111DB, 0x111DC, 0x111DD, 0x11200, 0x11212, 0x11213, 0x1122C, 0x1122F, 0x11232, 0x11234, + 0x11235, 0x11236, 0x11238, 0x1123E, 0x1123F, 0x11241, 0x11242, 0x11280, 0x11287, 0x11288, + 0x11289, 0x1128A, 0x1128E, 0x1128F, 0x1129E, 0x1129F, 0x112A9, 0x112B0, 0x112DF, 0x112E0, + 0x112E3, 0x112EB, 0x112F0, 0x112FA, 0x11300, 0x11302, 0x11304, 0x11305, 0x1130D, 0x1130F, + 0x11311, 0x11313, 0x11329, 0x1132A, 0x11331, 0x11332, 0x11334, 0x11335, 0x1133A, 0x1133B, + 0x1133D, 0x1133E, 0x11340, 0x11341, 0x11345, 0x11347, 0x11349, 0x1134B, 0x1134E, 0x11350, + 0x11351, 0x11357, 0x11358, 0x1135D, 0x11362, 0x11364, 0x11366, 0x1136D, 0x11370, 0x11375, + 0x11380, 0x1138A, 0x1138B, 0x1138C, 0x1138E, 0x1138F, 0x11390, 0x113B6, 0x113B7, 0x113B8, + 0x113BB, 0x113C1, 0x113C2, 0x113C3, 0x113C5, 0x113C6, 0x113C7, 0x113CB, 0x113CC, 0x113CE, + 0x113CF, 0x113D0, 0x113D1, 0x113D2, 0x113D3, 0x113D4, 0x113E1, 0x113E3, 0x11400, 0x11435, + 0x11438, 0x11440, 0x11442, 0x11445, 0x11446, 0x11447, 0x1144B, 0x11450, 0x1145A, 0x1145E, + 0x1145F, 0x11462, 0x11480, 0x114B0, 0x114B3, 0x114B9, 0x114BA, 0x114BB, 0x114BF, 0x114C1, + 0x114C2, 0x114C4, 0x114C6, 0x114C7, 0x114C8, 0x114D0, 0x114DA, 0x11580, 0x115AF, 0x115B2, + 0x115B6, 0x115B8, 0x115BC, 0x115BE, 0x115BF, 0x115C1, 0x115D8, 0x115DC, 0x115DE, 0x11600, + 0x11630, 0x11633, 0x1163B, 0x1163D, 0x1163E, 0x1163F, 0x11641, 0x11644, 0x11645, 0x11650, + 0x1165A, 0x11680, 0x116AB, 0x116AC, 0x116AD, 0x116AE, 0x116B0, 0x116B6, 0x116B7, 0x116B8, + 0x116B9, 0x116C0, 0x116CA, 0x116D0, 0x116E4, 0x1171D, 0x1171E, 0x1171F, 0x11720, 0x11722, + 0x11726, 0x11727, 0x1172C, 0x11730, 0x1173A, 0x11800, 0x1182C, 0x1182F, 0x11838, 0x11839, + 0x1183B, 0x118A0, 0x118E0, 0x118EA, 0x118FF, 0x11907, 0x11909, 0x1190A, 0x1190C, 0x11914, + 0x11915, 0x11917, 0x11918, 0x11930, 0x11936, 0x11937, 0x11939, 0x1193B, 0x1193D, 0x1193E, + 0x1193F, 0x11940, 0x11941, 0x11942, 0x11943, 0x11944, 0x11950, 0x1195A, 0x119A0, 0x119A8, + 0x119AA, 0x119D1, 0x119D4, 0x119D8, 0x119DA, 0x119DC, 0x119E0, 0x119E1, 0x119E2, 0x119E3, + 0x119E4, 0x119E5, 0x11A00, 0x11A01, 0x11A0B, 0x11A33, 0x11A39, 0x11A3A, 0x11A3B, 0x11A3F, + 0x11A47, 0x11A48, 0x11A50, 0x11A51, 0x11A57, 0x11A59, 0x11A5C, 0x11A8A, 0x11A97, 0x11A98, + 0x11A9A, 0x11A9D, 0x11A9E, 0x11AB0, 0x11AF9, 0x11B60, 0x11B61, 0x11B62, 0x11B65, 0x11B66, + 0x11B67, 0x11B68, 0x11BC0, 0x11BE1, 0x11BF0, 0x11BFA, 0x11C00, 0x11C09, 0x11C0A, 0x11C2F, + 0x11C30, 0x11C37, 0x11C38, 0x11C3E, 0x11C3F, 0x11C40, 0x11C41, 0x11C50, 0x11C5A, 0x11C72, + 0x11C90, 0x11C92, 0x11CA8, 0x11CA9, 0x11CAA, 0x11CB1, 0x11CB2, 0x11CB4, 0x11CB5, 0x11CB7, + 0x11D00, 0x11D07, 0x11D08, 0x11D0A, 0x11D0B, 0x11D31, 0x11D37, 0x11D3A, 0x11D3B, 0x11D3C, + 0x11D3E, 0x11D3F, 0x11D46, 0x11D47, 0x11D48, 0x11D50, 0x11D5A, 0x11D60, 0x11D66, 0x11D67, + 0x11D69, 0x11D6A, 0x11D8A, 0x11D8F, 0x11D90, 0x11D92, 0x11D93, 0x11D95, 0x11D96, 0x11D97, + 0x11D98, 0x11D99, 0x11DA0, 0x11DAA, 0x11DB0, 0x11DD9, 0x11DDA, 0x11DDC, 0x11DE0, 0x11DEA, + 0x11EE0, 0x11EF3, 0x11EF5, 0x11EF7, 0x11F00, 0x11F02, 0x11F03, 0x11F04, 0x11F11, 0x11F12, + 0x11F34, 0x11F36, 0x11F3B, 0x11F3E, 0x11F40, 0x11F41, 0x11F42, 0x11F43, 0x11F50, 0x11F5A, + 0x11F5B, 0x11FB0, 0x11FB1, 0x12000, 0x1239A, 0x12400, 0x1246F, 0x12480, 0x12544, 0x12F90, + 0x12FF1, 0x13000, 0x13430, 0x13440, 0x13441, 0x13447, 0x13456, 0x13460, 0x143FB, 0x14400, + 0x14647, 0x16100, 0x1611E, 0x1612A, 0x1612D, 0x16130, 0x1613A, 0x16800, 0x16A39, 0x16A40, + 0x16A5F, 0x16A60, 0x16A6A, 0x16A70, 0x16ABF, 0x16AC0, 0x16ACA, 0x16AD0, 0x16AEE, 0x16AF0, + 0x16AF5, 0x16B00, 0x16B30, 0x16B37, 0x16B40, 0x16B44, 0x16B50, 0x16B5A, 0x16B63, 0x16B78, + 0x16B7D, 0x16B90, 0x16D40, 0x16D43, 0x16D6B, 0x16D6D, 0x16D70, 0x16D7A, 0x16E40, 0x16E80, + 0x16EA0, 0x16EB9, 0x16EBB, 0x16ED4, 0x16F00, 0x16F4B, 0x16F4F, 0x16F50, 0x16F51, 0x16F88, + 0x16F8F, 0x16F93, 0x16FA0, 0x16FE0, 0x16FE2, 0x16FE3, 0x16FE4, 0x16FE5, 0x16FF0, 0x16FF2, + 0x1AFF0, 0x1AFF4, 0x1AFF5, 0x1AFFC, 0x1AFFD, 0x1AFFF, 0x1B000, 0x1B001, 0x1B120, 0x1B123, + 0x1B155, 0x1B156, 0x1B164, 0x1B168, 0x1BC00, 0x1BC6B, 0x1BC70, 0x1BC7D, 0x1BC80, 0x1BC89, + 0x1BC90, 0x1BC9A, 0x1BC9D, 0x1BC9F, 0x1BCA0, 0x1BCA4, 0x1CCF0, 0x1CCFA, 0x1CF00, 0x1CF2E, + 0x1CF30, 0x1CF47, 0x1D165, 0x1D167, 0x1D16A, 0x1D16D, 0x1D173, 0x1D17B, 0x1D183, 0x1D185, + 0x1D18C, 0x1D1AA, 0x1D1AE, 0x1D242, 0x1D245, 0x1D400, 0x1D455, 0x1D456, 0x1D49D, 0x1D49E, + 0x1D4A0, 0x1D4A2, 0x1D4A3, 0x1D4A5, 0x1D4A7, 0x1D4A9, 0x1D4AD, 0x1D4AE, 0x1D4BA, 0x1D4BB, + 0x1D4BC, 0x1D4BD, 0x1D4C4, 0x1D4C5, 0x1D506, 0x1D507, 0x1D50B, 0x1D50D, 0x1D515, 0x1D516, + 0x1D51D, 0x1D51E, 0x1D53A, 0x1D53B, 0x1D53F, 0x1D540, 0x1D545, 0x1D546, 0x1D547, 0x1D54A, + 0x1D551, 0x1D552, 0x1D6A6, 0x1D6A8, 0x1D6C1, 0x1D6C2, 0x1D6DB, 0x1D6DC, 0x1D6FB, 0x1D6FC, + 0x1D715, 0x1D716, 0x1D735, 0x1D736, 0x1D74F, 0x1D750, 0x1D76F, 0x1D770, 0x1D789, 0x1D78A, + 0x1D7A9, 0x1D7AA, 0x1D7C3, 0x1D7C4, 0x1D7CC, 0x1D7CE, 0x1D800, 0x1DA00, 0x1DA37, 0x1DA3B, + 0x1DA6D, 0x1DA75, 0x1DA76, 0x1DA84, 0x1DA85, 0x1DA9B, 0x1DAA0, 0x1DAA1, 0x1DAB0, 0x1DF00, + 0x1DF0A, 0x1DF0B, 0x1DF1F, 0x1DF25, 0x1DF2B, 0x1E000, 0x1E007, 0x1E008, 0x1E019, 0x1E01B, + 0x1E022, 0x1E023, 0x1E025, 0x1E026, 0x1E02B, 0x1E030, 0x1E06E, 0x1E08F, 0x1E090, 0x1E100, + 0x1E12D, 0x1E130, 0x1E137, 0x1E13E, 0x1E140, 0x1E14A, 0x1E14E, 0x1E14F, 0x1E290, 0x1E2AE, + 0x1E2AF, 0x1E2C0, 0x1E2EC, 0x1E2F0, 0x1E2FA, 0x1E4D0, 0x1E4EB, 0x1E4EC, 0x1E4F0, 0x1E4FA, + 0x1E5D0, 0x1E5EE, 0x1E5F0, 0x1E5F1, 0x1E5FB, 0x1E6C0, 0x1E6DF, 0x1E6E0, 0x1E6E3, 0x1E6E4, + 0x1E6E6, 0x1E6E7, 0x1E6EE, 0x1E6F0, 0x1E6F5, 0x1E6F6, 0x1E6FE, 0x1E6FF, 0x1E700, 0x1E7E0, + 0x1E7E7, 0x1E7E8, 0x1E7EC, 0x1E7ED, 0x1E7EF, 0x1E7F0, 0x1E7FF, 0x1E800, 0x1E8C5, 0x1E8D0, + 0x1E8D7, 0x1E900, 0x1E944, 0x1E94B, 0x1E94C, 0x1E950, 0x1E95A, 0x1EE00, 0x1EE04, 0x1EE05, + 0x1EE20, 0x1EE21, 0x1EE23, 0x1EE24, 0x1EE25, 0x1EE27, 0x1EE28, 0x1EE29, 0x1EE33, 0x1EE34, + 0x1EE38, 0x1EE39, 0x1EE3A, 0x1EE3B, 0x1EE3C, 0x1EE42, 0x1EE43, 0x1EE47, 0x1EE48, 0x1EE49, + 0x1EE4A, 0x1EE4B, 0x1EE4C, 0x1EE4D, 0x1EE50, 0x1EE51, 0x1EE53, 0x1EE54, 0x1EE55, 0x1EE57, + 0x1EE58, 0x1EE59, 0x1EE5A, 0x1EE5B, 0x1EE5C, 0x1EE5D, 0x1EE5E, 0x1EE5F, 0x1EE60, 0x1EE61, + 0x1EE63, 0x1EE64, 0x1EE65, 0x1EE67, 0x1EE6B, 0x1EE6C, 0x1EE73, 0x1EE74, 0x1EE78, 0x1EE79, + 0x1EE7D, 0x1EE7E, 0x1EE7F, 0x1EE80, 0x1EE8A, 0x1EE8B, 0x1EE9C, 0x1EEA1, 0x1EEA4, 0x1EEA5, + 0x1EEAA, 0x1EEAB, 0x1EEBC, 0x1F130, 0x1F14A, 0x1F150, 0x1F16A, 0x1F170, 0x1F18A, 0x1F1E6, + 0x1F200, 0x1F3FB, 0x1F400, 0x1FBF0, 0x1FBFA, 0xE0001, 0xE0002, 0xE0020, 0xE0080, 0xE0100, + 0xE01F0, + }; + + private static final byte[] VALUES = { + OTHER, LF, NEWLINE, CR, OTHER, WSEG_SPACE, OTHER, DOUBLE_QUOTE, OTHER, SINGLE_QUOTE, OTHER, MID_NUM, OTHER, MID_NUM_LET, OTHER, NUMERIC, + MID_LETTER, MID_NUM, OTHER, ALETTER, OTHER, EXTEND_NUM_LET, OTHER, ALETTER, OTHER, NEWLINE, OTHER, ALETTER, OTHER, FORMAT, OTHER, ALETTER, + OTHER, MID_LETTER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, + ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, OTHER, ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, EXTEND, ALETTER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, ALETTER, MID_NUM, ALETTER, OTHER, ALETTER, MID_LETTER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, ALETTER, OTHER, EXTEND, EXTEND, ALETTER, OTHER, ALETTER, OTHER, ALETTER, ALETTER, OTHER, ALETTER, MID_LETTER, ALETTER, + MID_NUM, ALETTER, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, HEBREW_LETTER, OTHER, HEBREW_LETTER, + ALETTER, MID_LETTER, OTHER, NUMERIC, OTHER, MID_NUM, OTHER, EXTEND, OTHER, FORMAT, OTHER, ALETTER, ALETTER, ALETTER, EXTEND, NUMERIC, + OTHER, NUMERIC, MID_NUM, OTHER, ALETTER, EXTEND, ALETTER, OTHER, ALETTER, EXTEND, NUMERIC, OTHER, EXTEND, ALETTER, EXTEND, OTHER, + EXTEND, ALETTER, NUMERIC, ALETTER, OTHER, ALETTER, OTHER, ALETTER, ALETTER, EXTEND, ALETTER, EXTEND, OTHER, ALETTER, EXTEND, ALETTER, + OTHER, NUMERIC, ALETTER, EXTEND, ALETTER, OTHER, MID_NUM, OTHER, ALETTER, OTHER, EXTEND, OTHER, ALETTER, EXTEND, ALETTER, EXTEND, + ALETTER, EXTEND, ALETTER, EXTEND, OTHER, ALETTER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, NUMERIC, OTHER, EXTEND, + ALETTER, ALETTER, EXTEND, NUMERIC, EXTEND, EXTEND, ALETTER, EXTEND, EXTEND, EXTEND, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, + ALETTER, EXTEND, ALETTER, EXTEND, OTHER, NUMERIC, OTHER, ALETTER, ALETTER, EXTEND, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, ALETTER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, + EXTEND, ALETTER, OTHER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, EXTEND, OTHER, NUMERIC, ALETTER, OTHER, ALETTER, OTHER, EXTEND, + OTHER, EXTEND, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, EXTEND, OTHER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, + OTHER, NUMERIC, EXTEND, ALETTER, EXTEND, OTHER, EXTEND, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, ALETTER, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, OTHER, + ALETTER, OTHER, ALETTER, EXTEND, OTHER, NUMERIC, OTHER, ALETTER, EXTEND, OTHER, EXTEND, EXTEND, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, + EXTEND, OTHER, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, OTHER, ALETTER, OTHER, ALETTER, EXTEND, OTHER, NUMERIC, OTHER, ALETTER, + OTHER, EXTEND, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, OTHER, ALETTER, + OTHER, EXTEND, OTHER, NUMERIC, OTHER, EXTEND, EXTEND, EXTEND, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + EXTEND, ALETTER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + EXTEND, OTHER, NUMERIC, OTHER, ALETTER, EXTEND, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, EXTEND, ALETTER, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, + ALETTER, OTHER, ALETTER, EXTEND, OTHER, NUMERIC, OTHER, ALETTER, EXTEND, OTHER, EXTEND, EXTEND, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, EXTEND, ALETTER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, ALETTER, OTHER, ALETTER, EXTEND, OTHER, ALETTER, + EXTEND, OTHER, NUMERIC, OTHER, ALETTER, OTHER, EXTEND, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, EXTEND, OTHER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, NUMERIC, OTHER, EXTEND, OTHER, + EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, NUMERIC, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, NUMERIC, OTHER, + ALETTER, OTHER, EXTEND, OTHER, NUMERIC, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, ALETTER, OTHER, ALETTER, + OTHER, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, ALETTER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, + EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, NUMERIC, OTHER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, + OTHER, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, NUMERIC, EXTEND, EXTEND, OTHER, ALETTER, OTHER, + ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, ALETTER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, ALETTER, WSEG_SPACE, ALETTER, OTHER, ALETTER, OTHER, ALETTER, ALETTER, OTHER, ALETTER, EXTEND, EXTEND, OTHER, ALETTER, + EXTEND, EXTEND, OTHER, ALETTER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, + EXTEND, EXTEND, EXTEND, OTHER, EXTEND, OTHER, NUMERIC, OTHER, EXTEND, FORMAT, EXTEND, NUMERIC, OTHER, ALETTER, ALETTER, ALETTER, + OTHER, ALETTER, EXTEND, ALETTER, EXTEND, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, + EXTEND, EXTEND, EXTEND, EXTEND, OTHER, NUMERIC, OTHER, NUMERIC, NUMERIC, OTHER, ALETTER, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, + EXTEND, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, NUMERIC, OTHER, NUMERIC, + OTHER, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, + EXTEND, EXTEND, ALETTER, OTHER, NUMERIC, OTHER, EXTEND, OTHER, EXTEND, EXTEND, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, + EXTEND, ALETTER, NUMERIC, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, ALETTER, EXTEND, EXTEND, + EXTEND, EXTEND, OTHER, NUMERIC, OTHER, ALETTER, NUMERIC, ALETTER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + EXTEND, OTHER, EXTEND, EXTEND, EXTEND, ALETTER, EXTEND, ALETTER, EXTEND, ALETTER, EXTEND, EXTEND, ALETTER, OTHER, ALETTER, ALETTER, + ALETTER, ALETTER, ALETTER, ALETTER, EXTEND, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, WSEG_SPACE, OTHER, WSEG_SPACE, OTHER, EXTEND, + ZWJ, FORMAT, OTHER, MID_NUM_LET, MID_NUM_LET, OTHER, MID_NUM_LET, OTHER, MID_LETTER, NEWLINE, NEWLINE, FORMAT, EXTEND_NUM_LET, OTHER, EXTEND_NUM_LET, OTHER, + MID_NUM, OTHER, EXTEND_NUM_LET, OTHER, WSEG_SPACE, FORMAT, OTHER, FORMAT, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, + EXTEND, EXTEND, EXTEND, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, ALETTER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, ALETTER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, ALETTER, ALETTER, OTHER, ALETTER, EXTEND, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, OTHER, ALETTER, OTHER, + WSEG_SPACE, OTHER, ALETTER, OTHER, EXTEND, EXTEND, OTHER, KATAKANA, OTHER, ALETTER, ALETTER, OTHER, EXTEND, KATAKANA, OTHER, KATAKANA, + KATAKANA, OTHER, KATAKANA, KATAKANA, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, KATAKANA, OTHER, KATAKANA, OTHER, KATAKANA, + OTHER, ALETTER, ALETTER, ALETTER, OTHER, ALETTER, ALETTER, OTHER, ALETTER, ALETTER, OTHER, ALETTER, NUMERIC, ALETTER, OTHER, ALETTER, + ALETTER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, ALETTER, ALETTER, ALETTER, EXTEND, ALETTER, ALETTER, EXTEND, OTHER, ALETTER, ALETTER, + ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, OTHER, ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, + EXTEND, ALETTER, EXTEND, ALETTER, EXTEND, ALETTER, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, OTHER, ALETTER, OTHER, EXTEND, ALETTER, + EXTEND, EXTEND, OTHER, NUMERIC, OTHER, EXTEND, ALETTER, OTHER, ALETTER, OTHER, ALETTER, EXTEND, NUMERIC, ALETTER, EXTEND, OTHER, + ALETTER, EXTEND, EXTEND, OTHER, ALETTER, OTHER, EXTEND, EXTEND, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, + ALETTER, NUMERIC, OTHER, EXTEND, OTHER, NUMERIC, OTHER, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, ALETTER, EXTEND, + ALETTER, EXTEND, EXTEND, OTHER, NUMERIC, OTHER, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, + EXTEND, OTHER, EXTEND, OTHER, ALETTER, EXTEND, EXTEND, EXTEND, OTHER, ALETTER, ALETTER, EXTEND, EXTEND, OTHER, ALETTER, OTHER, + ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, ALETTER, ALETTER, ALETTER, ALETTER, OTHER, ALETTER, ALETTER, + EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, OTHER, NUMERIC, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, HEBREW_LETTER, EXTEND, HEBREW_LETTER, OTHER, HEBREW_LETTER, OTHER, HEBREW_LETTER, OTHER, HEBREW_LETTER, OTHER, HEBREW_LETTER, + OTHER, HEBREW_LETTER, OTHER, HEBREW_LETTER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, OTHER, + MID_LETTER, OTHER, EXTEND, OTHER, EXTEND_NUM_LET, OTHER, EXTEND_NUM_LET, MID_NUM, OTHER, MID_NUM_LET, OTHER, MID_NUM, MID_LETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, FORMAT, OTHER, MID_NUM_LET, OTHER, MID_NUM, OTHER, MID_NUM_LET, OTHER, NUMERIC, MID_LETTER, MID_NUM, OTHER, ALETTER, OTHER, + EXTEND_NUM_LET, OTHER, ALETTER, OTHER, KATAKANA, KATAKANA, KATAKANA, EXTEND, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, FORMAT, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, ALETTER, OTHER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, ALETTER, + ALETTER, ALETTER, OTHER, ALETTER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, ALETTER, + OTHER, NUMERIC, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, EXTEND, OTHER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, EXTEND, OTHER, + ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, EXTEND, + OTHER, NUMERIC, OTHER, NUMERIC, ALETTER, ALETTER, ALETTER, ALETTER, OTHER, EXTEND, OTHER, ALETTER, ALETTER, OTHER, ALETTER, OTHER, + EXTEND, OTHER, ALETTER, OTHER, ALETTER, ALETTER, ALETTER, OTHER, EXTEND, ALETTER, OTHER, ALETTER, OTHER, ALETTER, EXTEND, OTHER, + ALETTER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, EXTEND, EXTEND, ALETTER, EXTEND, OTHER, NUMERIC, EXTEND, ALETTER, + EXTEND, ALETTER, OTHER, EXTEND, EXTEND, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, NUMERIC, OTHER, EXTEND, OTHER, NUMERIC, + OTHER, ALETTER, OTHER, NUMERIC, OTHER, EXTEND, ALETTER, EXTEND, EXTEND, EXTEND, OTHER, NUMERIC, OTHER, ALETTER, EXTEND, ALETTER, + OTHER, ALETTER, EXTEND, OTHER, ALETTER, OTHER, EXTEND, EXTEND, ALETTER, EXTEND, EXTEND, EXTEND, ALETTER, OTHER, EXTEND, OTHER, + EXTEND, EXTEND, NUMERIC, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, + OTHER, EXTEND, ALETTER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + EXTEND, EXTEND, EXTEND, OTHER, NUMERIC, OTHER, EXTEND, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, ALETTER, EXTEND, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, ALETTER, + OTHER, EXTEND, OTHER, ALETTER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, ALETTER, EXTEND, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, + ALETTER, EXTEND, ALETTER, OTHER, EXTEND, OTHER, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, ALETTER, OTHER, NUMERIC, + OTHER, EXTEND, ALETTER, OTHER, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, ALETTER, OTHER, ALETTER, + OTHER, NUMERIC, OTHER, ALETTER, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, ALETTER, EXTEND, OTHER, ALETTER, + EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, ALETTER, OTHER, NUMERIC, OTHER, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, + EXTEND, EXTEND, EXTEND, ALETTER, OTHER, NUMERIC, OTHER, NUMERIC, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, + OTHER, NUMERIC, OTHER, ALETTER, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, ALETTER, NUMERIC, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, OTHER, ALETTER, OTHER, ALETTER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, ALETTER, EXTEND, ALETTER, EXTEND, + EXTEND, OTHER, NUMERIC, OTHER, ALETTER, OTHER, ALETTER, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, ALETTER, OTHER, ALETTER, + EXTEND, OTHER, ALETTER, EXTEND, ALETTER, EXTEND, EXTEND, ALETTER, EXTEND, OTHER, EXTEND, OTHER, ALETTER, EXTEND, EXTEND, EXTEND, + ALETTER, EXTEND, EXTEND, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, + ALETTER, OTHER, NUMERIC, OTHER, ALETTER, OTHER, ALETTER, EXTEND, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, ALETTER, OTHER, NUMERIC, + OTHER, ALETTER, OTHER, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, + ALETTER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, ALETTER, EXTEND, OTHER, NUMERIC, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, EXTEND, EXTEND, ALETTER, OTHER, NUMERIC, OTHER, ALETTER, ALETTER, + ALETTER, OTHER, NUMERIC, OTHER, ALETTER, EXTEND, EXTEND, OTHER, EXTEND, ALETTER, EXTEND, ALETTER, OTHER, ALETTER, EXTEND, EXTEND, + OTHER, EXTEND, EXTEND, EXTEND, EXTEND, OTHER, NUMERIC, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, FORMAT, EXTEND, ALETTER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, EXTEND, EXTEND, + EXTEND, NUMERIC, OTHER, ALETTER, OTHER, ALETTER, OTHER, NUMERIC, OTHER, ALETTER, OTHER, NUMERIC, OTHER, ALETTER, OTHER, EXTEND, + OTHER, ALETTER, EXTEND, OTHER, ALETTER, OTHER, NUMERIC, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, ALETTER, ALETTER, OTHER, + NUMERIC, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, ALETTER, EXTEND, OTHER, EXTEND, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, EXTEND, OTHER, EXTEND, OTHER, KATAKANA, OTHER, KATAKANA, OTHER, KATAKANA, OTHER, KATAKANA, OTHER, + KATAKANA, OTHER, KATAKANA, OTHER, KATAKANA, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, OTHER, + FORMAT, OTHER, NUMERIC, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, EXTEND, OTHER, EXTEND, FORMAT, EXTEND, OTHER, EXTEND, + OTHER, EXTEND, OTHER, EXTEND, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, NUMERIC, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, ALETTER, + ALETTER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, EXTEND, OTHER, ALETTER, + OTHER, EXTEND, OTHER, ALETTER, OTHER, EXTEND, ALETTER, OTHER, NUMERIC, OTHER, ALETTER, OTHER, ALETTER, EXTEND, OTHER, ALETTER, + EXTEND, NUMERIC, OTHER, ALETTER, ALETTER, EXTEND, NUMERIC, OTHER, ALETTER, EXTEND, ALETTER, NUMERIC, OTHER, ALETTER, OTHER, ALETTER, + EXTEND, ALETTER, EXTEND, ALETTER, EXTEND, ALETTER, EXTEND, OTHER, ALETTER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, EXTEND, OTHER, ALETTER, EXTEND, ALETTER, OTHER, NUMERIC, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, OTHER, ALETTER, + OTHER, ALETTER, OTHER, ALETTER, OTHER, REGIONAL_INDICATOR, OTHER, EXTEND, OTHER, NUMERIC, OTHER, FORMAT, OTHER, EXTEND, OTHER, EXTEND, + OTHER, + }; + + private static final int[] EXTENDED_PICTOGRAPHIC_RANGES = { + 0xA9, 0xA9, 0xAE, 0xAE, 0x203C, 0x203C, 0x2049, 0x2049, + 0x2122, 0x2122, 0x2139, 0x2139, 0x2194, 0x2199, 0x21A9, 0x21AA, + 0x231A, 0x231B, 0x2328, 0x2328, 0x23CF, 0x23CF, 0x23E9, 0x23EC, + 0x23ED, 0x23EE, 0x23EF, 0x23EF, 0x23F0, 0x23F0, 0x23F1, 0x23F2, + 0x23F3, 0x23F3, 0x23F8, 0x23FA, 0x24C2, 0x24C2, 0x25AA, 0x25AB, + 0x25B6, 0x25B6, 0x25C0, 0x25C0, 0x25FB, 0x25FE, 0x2600, 0x2601, + 0x2602, 0x2603, 0x2604, 0x2604, 0x260E, 0x260E, 0x2611, 0x2611, + 0x2614, 0x2615, 0x2618, 0x2618, 0x261D, 0x261D, 0x2620, 0x2620, + 0x2622, 0x2623, 0x2626, 0x2626, 0x262A, 0x262A, 0x262E, 0x262E, + 0x262F, 0x262F, 0x2638, 0x2639, 0x263A, 0x263A, 0x2640, 0x2640, + 0x2642, 0x2642, 0x2648, 0x2653, 0x265F, 0x265F, 0x2660, 0x2660, + 0x2663, 0x2663, 0x2665, 0x2666, 0x2668, 0x2668, 0x267B, 0x267B, + 0x267E, 0x267E, 0x267F, 0x267F, 0x2692, 0x2692, 0x2693, 0x2693, + 0x2694, 0x2694, 0x2695, 0x2695, 0x2696, 0x2697, 0x2699, 0x2699, + 0x269B, 0x269C, 0x26A0, 0x26A1, 0x26A7, 0x26A7, 0x26AA, 0x26AB, + 0x26B0, 0x26B1, 0x26BD, 0x26BE, 0x26C4, 0x26C5, 0x26C8, 0x26C8, + 0x26CE, 0x26CE, 0x26CF, 0x26CF, 0x26D1, 0x26D1, 0x26D3, 0x26D3, + 0x26D4, 0x26D4, 0x26E9, 0x26E9, 0x26EA, 0x26EA, 0x26F0, 0x26F1, + 0x26F2, 0x26F3, 0x26F4, 0x26F4, 0x26F5, 0x26F5, 0x26F7, 0x26F9, + 0x26FA, 0x26FA, 0x26FD, 0x26FD, 0x2702, 0x2702, 0x2705, 0x2705, + 0x2708, 0x270C, 0x270D, 0x270D, 0x270F, 0x270F, 0x2712, 0x2712, + 0x2714, 0x2714, 0x2716, 0x2716, 0x271D, 0x271D, 0x2721, 0x2721, + 0x2728, 0x2728, 0x2733, 0x2734, 0x2744, 0x2744, 0x2747, 0x2747, + 0x274C, 0x274C, 0x274E, 0x274E, 0x2753, 0x2755, 0x2757, 0x2757, + 0x2763, 0x2763, 0x2764, 0x2764, 0x2795, 0x2797, 0x27A1, 0x27A1, + 0x27B0, 0x27B0, 0x27BF, 0x27BF, 0x2934, 0x2935, 0x2B05, 0x2B07, + 0x2B1B, 0x2B1C, 0x2B50, 0x2B50, 0x2B55, 0x2B55, 0x3030, 0x3030, + 0x303D, 0x303D, 0x3297, 0x3297, 0x3299, 0x3299, 0x1F004, 0x1F004, + 0x1F02C, 0x1F02F, 0x1F094, 0x1F09F, 0x1F0AF, 0x1F0B0, 0x1F0C0, 0x1F0C0, + 0x1F0CF, 0x1F0CF, 0x1F0D0, 0x1F0D0, 0x1F0F6, 0x1F0FF, 0x1F170, 0x1F171, + 0x1F17E, 0x1F17F, 0x1F18E, 0x1F18E, 0x1F191, 0x1F19A, 0x1F1AE, 0x1F1E5, + 0x1F201, 0x1F202, 0x1F203, 0x1F20F, 0x1F21A, 0x1F21A, 0x1F22F, 0x1F22F, + 0x1F232, 0x1F23A, 0x1F23C, 0x1F23F, 0x1F249, 0x1F24F, 0x1F250, 0x1F251, + 0x1F252, 0x1F25F, 0x1F266, 0x1F2FF, 0x1F300, 0x1F30C, 0x1F30D, 0x1F30E, + 0x1F30F, 0x1F30F, 0x1F310, 0x1F310, 0x1F311, 0x1F311, 0x1F312, 0x1F312, + 0x1F313, 0x1F315, 0x1F316, 0x1F318, 0x1F319, 0x1F319, 0x1F31A, 0x1F31A, + 0x1F31B, 0x1F31B, 0x1F31C, 0x1F31C, 0x1F31D, 0x1F31E, 0x1F31F, 0x1F320, + 0x1F321, 0x1F321, 0x1F324, 0x1F32C, 0x1F32D, 0x1F32F, 0x1F330, 0x1F331, + 0x1F332, 0x1F333, 0x1F334, 0x1F335, 0x1F336, 0x1F336, 0x1F337, 0x1F34A, + 0x1F34B, 0x1F34B, 0x1F34C, 0x1F34F, 0x1F350, 0x1F350, 0x1F351, 0x1F37B, + 0x1F37C, 0x1F37C, 0x1F37D, 0x1F37D, 0x1F37E, 0x1F37F, 0x1F380, 0x1F393, + 0x1F396, 0x1F397, 0x1F399, 0x1F39B, 0x1F39E, 0x1F39F, 0x1F3A0, 0x1F3C4, + 0x1F3C5, 0x1F3C5, 0x1F3C6, 0x1F3C6, 0x1F3C7, 0x1F3C7, 0x1F3C8, 0x1F3C8, + 0x1F3C9, 0x1F3C9, 0x1F3CA, 0x1F3CA, 0x1F3CB, 0x1F3CE, 0x1F3CF, 0x1F3D3, + 0x1F3D4, 0x1F3DF, 0x1F3E0, 0x1F3E3, 0x1F3E4, 0x1F3E4, 0x1F3E5, 0x1F3F0, + 0x1F3F3, 0x1F3F3, 0x1F3F4, 0x1F3F4, 0x1F3F5, 0x1F3F5, 0x1F3F7, 0x1F3F7, + 0x1F3F8, 0x1F3FA, 0x1F400, 0x1F407, 0x1F408, 0x1F408, 0x1F409, 0x1F40B, + 0x1F40C, 0x1F40E, 0x1F40F, 0x1F410, 0x1F411, 0x1F412, 0x1F413, 0x1F413, + 0x1F414, 0x1F414, 0x1F415, 0x1F415, 0x1F416, 0x1F416, 0x1F417, 0x1F429, + 0x1F42A, 0x1F42A, 0x1F42B, 0x1F43E, 0x1F43F, 0x1F43F, 0x1F440, 0x1F440, + 0x1F441, 0x1F441, 0x1F442, 0x1F464, 0x1F465, 0x1F465, 0x1F466, 0x1F46B, + 0x1F46C, 0x1F46D, 0x1F46E, 0x1F4AC, 0x1F4AD, 0x1F4AD, 0x1F4AE, 0x1F4B5, + 0x1F4B6, 0x1F4B7, 0x1F4B8, 0x1F4EB, 0x1F4EC, 0x1F4ED, 0x1F4EE, 0x1F4EE, + 0x1F4EF, 0x1F4EF, 0x1F4F0, 0x1F4F4, 0x1F4F5, 0x1F4F5, 0x1F4F6, 0x1F4F7, + 0x1F4F8, 0x1F4F8, 0x1F4F9, 0x1F4FC, 0x1F4FD, 0x1F4FD, 0x1F4FF, 0x1F502, + 0x1F503, 0x1F503, 0x1F504, 0x1F507, 0x1F508, 0x1F508, 0x1F509, 0x1F509, + 0x1F50A, 0x1F514, 0x1F515, 0x1F515, 0x1F516, 0x1F52B, 0x1F52C, 0x1F52D, + 0x1F52E, 0x1F53D, 0x1F549, 0x1F54A, 0x1F54B, 0x1F54E, 0x1F550, 0x1F55B, + 0x1F55C, 0x1F567, 0x1F56F, 0x1F570, 0x1F573, 0x1F579, 0x1F57A, 0x1F57A, + 0x1F587, 0x1F587, 0x1F58A, 0x1F58D, 0x1F590, 0x1F590, 0x1F595, 0x1F596, + 0x1F5A4, 0x1F5A4, 0x1F5A5, 0x1F5A5, 0x1F5A8, 0x1F5A8, 0x1F5B1, 0x1F5B2, + 0x1F5BC, 0x1F5BC, 0x1F5C2, 0x1F5C4, 0x1F5D1, 0x1F5D3, 0x1F5DC, 0x1F5DE, + 0x1F5E1, 0x1F5E1, 0x1F5E3, 0x1F5E3, 0x1F5E8, 0x1F5E8, 0x1F5EF, 0x1F5EF, + 0x1F5F3, 0x1F5F3, 0x1F5FA, 0x1F5FA, 0x1F5FB, 0x1F5FF, 0x1F600, 0x1F600, + 0x1F601, 0x1F606, 0x1F607, 0x1F608, 0x1F609, 0x1F60D, 0x1F60E, 0x1F60E, + 0x1F60F, 0x1F60F, 0x1F610, 0x1F610, 0x1F611, 0x1F611, 0x1F612, 0x1F614, + 0x1F615, 0x1F615, 0x1F616, 0x1F616, 0x1F617, 0x1F617, 0x1F618, 0x1F618, + 0x1F619, 0x1F619, 0x1F61A, 0x1F61A, 0x1F61B, 0x1F61B, 0x1F61C, 0x1F61E, + 0x1F61F, 0x1F61F, 0x1F620, 0x1F625, 0x1F626, 0x1F627, 0x1F628, 0x1F62B, + 0x1F62C, 0x1F62C, 0x1F62D, 0x1F62D, 0x1F62E, 0x1F62F, 0x1F630, 0x1F633, + 0x1F634, 0x1F634, 0x1F635, 0x1F635, 0x1F636, 0x1F636, 0x1F637, 0x1F640, + 0x1F641, 0x1F644, 0x1F645, 0x1F64F, 0x1F680, 0x1F680, 0x1F681, 0x1F682, + 0x1F683, 0x1F685, 0x1F686, 0x1F686, 0x1F687, 0x1F687, 0x1F688, 0x1F688, + 0x1F689, 0x1F689, 0x1F68A, 0x1F68B, 0x1F68C, 0x1F68C, 0x1F68D, 0x1F68D, + 0x1F68E, 0x1F68E, 0x1F68F, 0x1F68F, 0x1F690, 0x1F690, 0x1F691, 0x1F693, + 0x1F694, 0x1F694, 0x1F695, 0x1F695, 0x1F696, 0x1F696, 0x1F697, 0x1F697, + 0x1F698, 0x1F698, 0x1F699, 0x1F69A, 0x1F69B, 0x1F6A1, 0x1F6A2, 0x1F6A2, + 0x1F6A3, 0x1F6A3, 0x1F6A4, 0x1F6A5, 0x1F6A6, 0x1F6A6, 0x1F6A7, 0x1F6AD, + 0x1F6AE, 0x1F6B1, 0x1F6B2, 0x1F6B2, 0x1F6B3, 0x1F6B5, 0x1F6B6, 0x1F6B6, + 0x1F6B7, 0x1F6B8, 0x1F6B9, 0x1F6BE, 0x1F6BF, 0x1F6BF, 0x1F6C0, 0x1F6C0, + 0x1F6C1, 0x1F6C5, 0x1F6CB, 0x1F6CB, 0x1F6CC, 0x1F6CC, 0x1F6CD, 0x1F6CF, + 0x1F6D0, 0x1F6D0, 0x1F6D1, 0x1F6D2, 0x1F6D5, 0x1F6D5, 0x1F6D6, 0x1F6D7, + 0x1F6D8, 0x1F6D8, 0x1F6D9, 0x1F6DB, 0x1F6DC, 0x1F6DC, 0x1F6DD, 0x1F6DF, + 0x1F6E0, 0x1F6E5, 0x1F6E9, 0x1F6E9, 0x1F6EB, 0x1F6EC, 0x1F6ED, 0x1F6EF, + 0x1F6F0, 0x1F6F0, 0x1F6F3, 0x1F6F3, 0x1F6F4, 0x1F6F6, 0x1F6F7, 0x1F6F8, + 0x1F6F9, 0x1F6F9, 0x1F6FA, 0x1F6FA, 0x1F6FB, 0x1F6FC, 0x1F6FD, 0x1F6FF, + 0x1F7DA, 0x1F7DF, 0x1F7E0, 0x1F7EB, 0x1F7EC, 0x1F7EF, 0x1F7F0, 0x1F7F0, + 0x1F7F1, 0x1F7FF, 0x1F80C, 0x1F80F, 0x1F848, 0x1F84F, 0x1F85A, 0x1F85F, + 0x1F888, 0x1F88F, 0x1F8AE, 0x1F8AF, 0x1F8BC, 0x1F8BF, 0x1F8C2, 0x1F8CF, + 0x1F8D9, 0x1F8FF, 0x1F90C, 0x1F90C, 0x1F90D, 0x1F90F, 0x1F910, 0x1F918, + 0x1F919, 0x1F91E, 0x1F91F, 0x1F91F, 0x1F920, 0x1F927, 0x1F928, 0x1F92F, + 0x1F930, 0x1F930, 0x1F931, 0x1F932, 0x1F933, 0x1F93A, 0x1F93C, 0x1F93E, + 0x1F93F, 0x1F93F, 0x1F940, 0x1F945, 0x1F947, 0x1F94B, 0x1F94C, 0x1F94C, + 0x1F94D, 0x1F94F, 0x1F950, 0x1F95E, 0x1F95F, 0x1F96B, 0x1F96C, 0x1F970, + 0x1F971, 0x1F971, 0x1F972, 0x1F972, 0x1F973, 0x1F976, 0x1F977, 0x1F978, + 0x1F979, 0x1F979, 0x1F97A, 0x1F97A, 0x1F97B, 0x1F97B, 0x1F97C, 0x1F97F, + 0x1F980, 0x1F984, 0x1F985, 0x1F991, 0x1F992, 0x1F997, 0x1F998, 0x1F9A2, + 0x1F9A3, 0x1F9A4, 0x1F9A5, 0x1F9AA, 0x1F9AB, 0x1F9AD, 0x1F9AE, 0x1F9AF, + 0x1F9B0, 0x1F9B9, 0x1F9BA, 0x1F9BF, 0x1F9C0, 0x1F9C0, 0x1F9C1, 0x1F9C2, + 0x1F9C3, 0x1F9CA, 0x1F9CB, 0x1F9CB, 0x1F9CC, 0x1F9CC, 0x1F9CD, 0x1F9CF, + 0x1F9D0, 0x1F9E6, 0x1F9E7, 0x1F9FF, 0x1FA58, 0x1FA5F, 0x1FA6E, 0x1FA6F, + 0x1FA70, 0x1FA73, 0x1FA74, 0x1FA74, 0x1FA75, 0x1FA77, 0x1FA78, 0x1FA7A, + 0x1FA7B, 0x1FA7C, 0x1FA7D, 0x1FA7F, 0x1FA80, 0x1FA82, 0x1FA83, 0x1FA86, + 0x1FA87, 0x1FA88, 0x1FA89, 0x1FA89, 0x1FA8A, 0x1FA8A, 0x1FA8B, 0x1FA8D, + 0x1FA8E, 0x1FA8E, 0x1FA8F, 0x1FA8F, 0x1FA90, 0x1FA95, 0x1FA96, 0x1FAA8, + 0x1FAA9, 0x1FAAC, 0x1FAAD, 0x1FAAF, 0x1FAB0, 0x1FAB6, 0x1FAB7, 0x1FABA, + 0x1FABB, 0x1FABD, 0x1FABE, 0x1FABE, 0x1FABF, 0x1FABF, 0x1FAC0, 0x1FAC2, + 0x1FAC3, 0x1FAC5, 0x1FAC6, 0x1FAC6, 0x1FAC7, 0x1FAC7, 0x1FAC8, 0x1FAC8, + 0x1FAC9, 0x1FACC, 0x1FACD, 0x1FACD, 0x1FACE, 0x1FACF, 0x1FAD0, 0x1FAD6, + 0x1FAD7, 0x1FAD9, 0x1FADA, 0x1FADB, 0x1FADC, 0x1FADC, 0x1FADD, 0x1FADE, + 0x1FADF, 0x1FADF, 0x1FAE0, 0x1FAE7, 0x1FAE8, 0x1FAE8, 0x1FAE9, 0x1FAE9, + 0x1FAEA, 0x1FAEA, 0x1FAEB, 0x1FAEE, 0x1FAEF, 0x1FAEF, 0x1FAF0, 0x1FAF6, + 0x1FAF7, 0x1FAF8, 0x1FAF9, 0x1FAFF, 0x1FC00, 0x1FFFD, + }; + + static byte propertyOf(int codePoint) { + int low = 0; + int high = STARTS.length - 1; + while (low <= high) { + int middle = (low + high) >>> 1; + if (STARTS[middle] <= codePoint) { + low = middle + 1; + } else { + high = middle - 1; + } + } + return VALUES[Math.max(0, high)]; + } + + static boolean isExtendedPictographic(int codePoint) { + int low = 0; + int high = EXTENDED_PICTOGRAPHIC_RANGES.length / 2 - 1; + while (low <= high) { + int middle = (low + high) >>> 1; + int start = EXTENDED_PICTOGRAPHIC_RANGES[middle * 2]; + int rangeEnd = EXTENDED_PICTOGRAPHIC_RANGES[middle * 2 + 1]; + if (codePoint < start) { + high = middle - 1; + } else if (codePoint > rangeEnd) { + low = middle + 1; + } else { + return true; + } + } + return false; + } + + private WordBreakData() { + } +} diff --git a/third_party/joni/src/org/joni/constants/internal/AnchorType.java b/third_party/joni/src/org/joni/constants/internal/AnchorType.java index e329d0cf29..f1cb3b99ef 100644 --- a/third_party/joni/src/org/joni/constants/internal/AnchorType.java +++ b/third_party/joni/src/org/joni/constants/internal/AnchorType.java @@ -47,6 +47,8 @@ public interface AnchorType { int NOT_GRAPHEME_BOUNDARY = (1<<18); int SENTENCE_BOUNDARY = (1<<19); int NOT_SENTENCE_BOUNDARY = (1<<20); + int WORD_BREAK_BOUNDARY = (1<<21); + int NOT_WORD_BREAK_BOUNDARY = (1<<22); int ALLOWED_IN_LB = ( LOOK_BEHIND | LOOK_BEHIND_NOT | @@ -61,6 +63,8 @@ public interface AnchorType { NOT_GRAPHEME_BOUNDARY | SENTENCE_BOUNDARY | NOT_SENTENCE_BOUNDARY | + WORD_BREAK_BOUNDARY | + NOT_WORD_BREAK_BOUNDARY | WORD_BEGIN | WORD_END ); @@ -78,6 +82,8 @@ public interface AnchorType { NOT_GRAPHEME_BOUNDARY | SENTENCE_BOUNDARY | NOT_SENTENCE_BOUNDARY | + WORD_BREAK_BOUNDARY | + NOT_WORD_BREAK_BOUNDARY | WORD_BEGIN | WORD_END ); } diff --git a/third_party/joni/src/org/joni/constants/internal/OPCode.java b/third_party/joni/src/org/joni/constants/internal/OPCode.java index 7b649ab2dd..fdec60efe3 100644 --- a/third_party/joni/src/org/joni/constants/internal/OPCode.java +++ b/third_party/joni/src/org/joni/constants/internal/OPCode.java @@ -158,6 +158,8 @@ public interface OPCode { int NOT_GRAPHEME_BOUNDARY = 114; /* Perl \B{gcb} */ int SENTENCE_BOUNDARY = 115; /* Perl \b{sb} */ int NOT_SENTENCE_BOUNDARY = 116; /* Perl \B{sb} */ + int WORD_BREAK_BOUNDARY = 117; /* Perl \b{wb} */ + int NOT_WORD_BREAK_BOUNDARY = 118; /* Perl \B{wb} */ String[] OpCodeNames = Config.DEBUG_COMPILE ? new String[] { "finish", /*OP_FINISH*/ @@ -278,6 +280,8 @@ public interface OPCode { "not-grapheme-boundary", /*OP_NOT_GRAPHEME_BOUNDARY*/ "sentence-boundary", /*OP_SENTENCE_BOUNDARY*/ "not-sentence-boundary", /*OP_NOT_SENTENCE_BOUNDARY*/ + "word-break-boundary", /*OP_WORD_BREAK_BOUNDARY*/ + "not-word-break-boundary", /*OP_NOT_WORD_BREAK_BOUNDARY*/ } : null; int[] OpCodeArgTypes = Config.DEBUG_COMPILE ? new int[] { @@ -399,5 +403,7 @@ public interface OPCode { Arguments.NON, /*OP_NOT_GRAPHEME_BOUNDARY*/ Arguments.NON, /*OP_SENTENCE_BOUNDARY*/ Arguments.NON, /*OP_NOT_SENTENCE_BOUNDARY*/ + Arguments.NON, /*OP_WORD_BREAK_BOUNDARY*/ + Arguments.NON, /*OP_NOT_WORD_BREAK_BOUNDARY*/ } : null; } diff --git a/third_party/joni/src/org/joni/exception/ErrorMessages.java b/third_party/joni/src/org/joni/exception/ErrorMessages.java index 45d6589337..e7b4323cd4 100644 --- a/third_party/joni/src/org/joni/exception/ErrorMessages.java +++ b/third_party/joni/src/org/joni/exception/ErrorMessages.java @@ -67,6 +67,13 @@ public interface ErrorMessages extends org.jcodings.exception.ErrorMessages { "Sequence (?<... not terminated"; String PERL_PYTHON_NAMED_BACKREF_NOT_TERMINATED = "Sequence ?P=... not terminated"; + String PERL_ALPHA_ASSERTION_REQUIRES_COLON = + "'(*%n' requires a terminating ':'"; + String PERL_UNKNOWN_VERB_PATTERN = "Unknown verb pattern '%n'"; + String PERL_UNKNOWN_CONTROL_CONSTRUCT = + "Unknown '(*...)' construct '%n'"; + String PERL_UNTERMINATED_CONTROL_ARGUMENT = + "Unterminated '(*...' argument"; String PERL_CARET_MINUS_OPTION_NOT_RECOGNIZED = "Sequence (?^-...) not recognized"; String PERL_CARET_D_OPTION_NOT_RECOGNIZED = diff --git a/third_party/joni/test/org/joni/test/TestPerl.java b/third_party/joni/test/org/joni/test/TestPerl.java index 0ad090cde9..708ab071d9 100755 --- a/third_party/joni/test/org/joni/test/TestPerl.java +++ b/third_party/joni/test/org/joni/test/TestPerl.java @@ -76,6 +76,23 @@ public void test() throws Exception { xerrs("(?P=name", ErrorMessages.PERL_PYTHON_NAMED_BACKREF_NOT_TERMINATED); ns("[(?P)]", "z"); x2s("\\(\\?P", "(?P", 0, 6); + x2s("a(*pla:b)b", "ab", 0, 2); + ns("a(*pla:c)b", "ab"); + x2s("a(*plb:a)b", "ab", 0, 2); + ns("a(*plb:c)b", "ab"); + x2s("a(*nla:c)b", "ab", 0, 2); + ns("a(*nla:b)b", "ab"); + x2s("a(*nlb:c)b", "ab", 0, 2); + ns("a(*nlb:a)b", "ab"); + ns("(*atomic:a|ab)c", "abc"); + x2s("a(*pla:(*nla:c)b)b", "ab", 0, 2); + x2s("(*pla:)", "", 0, 0); + xerrs("(*PLA:a)", "Unknown verb pattern 'PLA'"); + xerrs("(*pla)", "'(*pla' requires a terminating ':'"); + xerrs("(*plx:a)", "Unknown '(*...)' construct 'plx'"); + xerrs("(*pla:a", ErrorMessages.PERL_UNTERMINATED_CONTROL_ARGUMENT); + ns("[(?*pla:)]", "z"); + x2s("\\(\\*pla:a\\)", "(*pla:a)", 0, 8); } @org.junit.Test(timeout = 5000) diff --git a/third_party/joni/test/org/joni/test/TestPerlWordBoundary.java b/third_party/joni/test/org/joni/test/TestPerlWordBoundary.java new file mode 100644 index 0000000000..143fbc6031 --- /dev/null +++ b/third_party/joni/test/org/joni/test/TestPerlWordBoundary.java @@ -0,0 +1,67 @@ +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni.test; + +import static org.junit.Assert.assertEquals; + +import java.nio.charset.StandardCharsets; + +import org.jcodings.specific.UTF8Encoding; +import org.joni.Option; +import org.joni.Regex; +import org.joni.Syntax; +import org.junit.Test; + +public class TestPerlWordBoundary { + private static int search(String pattern, String input) { + byte[] patternBytes = pattern.getBytes(StandardCharsets.UTF_8); + byte[] inputBytes = input.getBytes(StandardCharsets.UTF_8); + Regex regex = new Regex(patternBytes, 0, patternBytes.length, Option.NONE, + UTF8Encoding.INSTANCE, Syntax.PerlNG); + return regex.matcher(inputBytes).search(0, inputBytes.length, Option.NONE); + } + + private static void assertBoundary(String left, String right) { + assertEquals(0, search("\\A" + left + "\\b{wb}" + right + "\\z", left + right)); + } + + private static void assertNoBoundary(String left, String right) { + assertEquals(0, search("\\A" + left + "\\B{wb}" + right + "\\z", left + right)); + } + + @Test + public void implementsUnicodeWordBreakRules() { + assertEquals(0, search("\\A\\r\\B{wb}\\n\\z", "\r\n")); + assertBoundary("\r", "A"); + assertNoBoundary("‍", "🚀"); + assertBoundary("‍̈", "🚀"); + assertBoundary("‍", "✁"); + assertNoBoundary(" ", " "); + assertBoundary(" ̈", " "); + assertNoBoundary("a", "b"); + assertNoBoundary("a.", "b"); + assertNoBoundary("א\"", "ב"); + assertNoBoundary("1,", "2"); + assertNoBoundary("ア", "イ"); + assertNoBoundary("🇦", "🇧"); + assertBoundary("🇦🇧", "🇨"); + assertBoundary("a", ",b"); + } +} From a1d1d221e76b505e80ca0b51a3e8ccd594239f48 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:32 +0200 Subject: [PATCH 24/60] feat(regex): complete global retry and pinned Age properties (#1029) Squashed integration of PR #1029 at badbdfe3f. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 45 +- dev/tools/generate_perl_unicode_age_data.pl | 117 ++++ docs/reference/feature-matrix.md | 6 +- .../runtime/regex/JoniRegexPattern.java | 2 +- .../runtime/regex/PerlUnicodeAgeData.java | 518 ++++++++++++++++++ .../runtime/regex/RuntimeRegex.java | 148 +++-- .../runtime/regex/UnicodeResolver.java | 63 +-- .../runtime/regex/PerlUnicodeAgeDataTest.java | 20 + .../unit/regex/unicode_age_loose_pinned.t | 26 + .../unit/regex/zero_width_global_retry.t | 213 +++++++ 10 files changed, 1054 insertions(+), 104 deletions(-) create mode 100644 dev/tools/generate_perl_unicode_age_data.pl create mode 100644 src/main/java/org/perlonjava/runtime/regex/PerlUnicodeAgeData.java create mode 100644 src/test/java/org/perlonjava/runtime/regex/PerlUnicodeAgeDataTest.java create mode 100644 src/test/resources/unit/regex/unicode_age_loose_pinned.t create mode 100644 src/test/resources/unit/regex/zero_width_global_retry.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 7daa1b7196..64241b4d39 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,15 +224,16 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1027, stacked +The published integration stack is preserved through draft PR #1028, stacked on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the first 524 lines of retired Java-only preprocessor code. Every published slice has a warning-free combined `make` checkpoint. PR #1027 adds native sentence -boundaries. The current WIP integrates independently validated alpha assertion -aliases and native word boundaries, while global zero-width `/g` progression -and line boundaries remain parallel integration slices. +boundaries; PR #1028 adds independently validated alpha assertion aliases and +native word boundaries. The current WIP integrates corrected global zero-width +`/g` progression and pinned Perl 5.44 Unicode 17.0 Age properties while line +boundaries remain a parallel integration slice. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -267,10 +268,19 @@ Extended_Pictographic data. The 33-assertion focused oracle passes on system Perl, JVM, and interpreter, direct Joni exercises the same path, and generated chunk 10 passes 19,510/19,510 identically on both execution backends. Combined with the complete boundary chunk 05 and unchanged chunks 06–09, current -generated evidence is 132,578/407,367. A resource-contended +generated evidence was 132,578/407,367 before the current property slice. A resource-contended current-head refresh did not reproduce a complete exact JVM/interpreter pair, so it does not replace that accepted baseline. +`Age` now uses exact introduction-version sets and `In`/`Present_In` use +cumulative sets generated from the repository-pinned Perl 5.44 Unicode 17.0 +`DAge.txt`; `Unassigned`/`NA`, colon delimiters, wildcard-value spellings, and +Perl loose version aliases are covered without inheriting the host ICU Unicode +version. The focused oracle passes 14/14 on system Perl, JVM, and interpreter. +Stable chunk 01 validation improves from 30,194 to 30,705 passing assertions +identically on JVM and interpreter, with no numbered regression. This raises +current generated evidence by 511 to 133,089/407,367. + Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while ordinary and substitution callbacks execute, without misclassifying escaped or @@ -367,10 +377,10 @@ matcher-specific timeouts on both execution backends. non-ASCII substitutions under lexical `use bytes`. Upgraded, byte-backed, and compiled byte-backed patterns pass 12/12 on all runtimes, and the generated Unicode marker stage passes 4/4 on JVM and interpreter. - - [ ] Close `/g` same-position retry and capture semantics after a zero-width - first alternative. The standard-Perl omniholder reducer currently passes - 7/10 in all PerlOnJava modes even though the DBIx substitution contract - passes; this is independent of the Java-only alternative-reorder rewrite. + - [x] Closed `/g` same-position retry and capture semantics after a zero-width + first alternative (`0703725c8`, integrated as `402102446`). The focused + oracle passes 23/23, the raw omniholder reducer improves from 7/10 to 10/10 + in all six Java/Joni × JVM/interpreter modes, and DBIx::Simple remains 69/69. - [x] Phase 2: Conditions and backtracking-visible state (2026-08-17) - [x] Implemented executable callback conditions, control verbs including `(*MARK:NAME)`, and callback-visible recursive capture state in Joni. @@ -475,6 +485,10 @@ matcher-specific timeouts on both execution backends. reproducible Perl 5.44 Unicode 17.0 Word_Break and Extended_Pictographic tables. The focused oracle passes 33/33 and generated chunk 10 passes 19,510/19,510 on both execution backends. + - [x] Generated exact `Age` and cumulative `In`/`Present_In` sets from pinned + Perl 5.44 Unicode 17.0 data, including loose version, wildcard, and + unassigned aliases. The focused oracle passes 14/14 and chunk 01 gains 511 + assertions with no numbered regression. - [ ] Implement the native Joni line boundary algorithm, then close the remaining property and boundary failures before marking Phase 3 complete. @@ -513,16 +527,15 @@ matcher-specific timeouts on both execution backends. leftmost-capture and 20,000-character gates in all four backend modes. - [x] Retired the Java-only DBIx omniholder alternative reorder (`18e71a532`; 50 lines removed) after exact substitution and bundled - DBIx::Simple gates passed without changing the separately tracked raw `/g` - 7/10 result. + DBIx::Simple gates passed. The independent raw `/g` 7/10 progression gap is + now closed at 10/10 by the shared matcher-adapter fix above. - [ ] Phase 6: Integration and release ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1027. Publish the current - WIP integration branch containing alpha-assertion aliases, then integrate the - pending zero-width `/g` adapter commit after its non-overlapping gates are - complete. +1. Land review-ready PR #1024 and draft PRs #1025–#1028. Publish the current + post-#1028 WIP containing the completed zero-width `/g` adapter integration, + then rerun the combined forced-backend differential. 2. Continue boundary semantics natively in Joni with line breaks (0/205,380 in chunks 06–09). PerlOnJava4 owns the non-overlapping line slice; word boundaries are complete at 19,510/19,510 in chunk 10. Use pinned Perl 5.44 Unicode data and @@ -530,7 +543,7 @@ matcher-specific timeouts on both execution backends. boundary rewrite only after native parity. 3. Implement the remaining property clusters in measured order: Block, Script/Script_Extensions, Numeric_Value, Joining_Group, General_Category, - break-property values, and Age/In/Present_In. Preserve pinned Perl 5.44 + break-property values. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. Close underscored numeric escapes with focused standard-Perl gates. 4. Rerun the forced-Joni 80-file corpus on JVM and interpreter from the combined diff --git a/dev/tools/generate_perl_unicode_age_data.pl b/dev/tools/generate_perl_unicode_age_data.pl new file mode 100644 index 0000000000..0aeaea23b2 --- /dev/null +++ b/dev/tools/generate_perl_unicode_age_data.pl @@ -0,0 +1,117 @@ +#!/usr/bin/env perl +use strict; +use warnings; + +my $unicode_root = 'perl5/lib/unicore'; +my $expected_unicode_version = '17.0.0'; + +open my $version_fh, '<', "$unicode_root/version" + or die "Can't read Unicode version: $!\n"; +chomp(my $unicode_version = <$version_fh>); +close $version_fh; +die "Expected Unicode $expected_unicode_version, found $unicode_version\n" + unless $unicode_version eq $expected_unicode_version; + +open my $age_fh, '<', "$unicode_root/DAge.txt" + or die "Can't read Age data: $!\n"; +my (@versions, %version_index, @ranges); +while (<$age_fh>) { + next unless /^([0-9A-F]+)(?:\.\.([0-9A-F]+))?\s*;\s*([0-9]+\.[0-9]+)/; + my ($start, $end, $version) = (hex($1), defined($2) ? hex($2) : hex($1), $3); + if (!exists $version_index{$version}) { + $version_index{$version} = scalar @versions; + push @versions, $version; + } + if (@ranges && $ranges[-1][2] == $version_index{$version} + && $ranges[-1][1] + 1 == $start) { + $ranges[-1][1] = $end; + } else { + push @ranges, [$start, $end, $version_index{$version}]; + } +} +close $age_fh; + +die "Age data does not end at Unicode 17.0\n" + unless @versions && $versions[-1] eq '17.0'; + +print <<'HEADER'; +/* + * Generated from Perl 5.44's Unicode Character Database. Do not edit manually. + * + * Unicode data source copyright: + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in + * the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + */ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; + +final class PerlUnicodeAgeData { +HEADER + +print " static final String UNICODE_VERSION = \"$unicode_version\";\n"; +print " private static final String[] VERSIONS = {\n "; +print join(', ', map { qq{"$_"} } @versions); +print "\n };\n\n"; +print " private static final int[] RANGES = {\n"; +for (my $i = 0; $i < @ranges; $i += 4) { + my $end = $i + 3 < $#ranges ? $i + 3 : $#ranges; + print " ", join(', ', map { + sprintf '0x%X, 0x%X, %d', @{$ranges[$_]}[0, 1, 2] + } $i .. $end), ",\n"; +} +print <<'FOOTER'; + }; + + private static final UnicodeSet[] EXACT_SETS = buildExactSets(); + private static final UnicodeSet[] CUMULATIVE_SETS = buildCumulativeSets(); + private static final UnicodeSet UNASSIGNED = new UnicodeSet(0, 0x10ffff) + .removeAll(CUMULATIVE_SETS[CUMULATIVE_SETS.length - 1]).freeze(); + + static UnicodeSet exactSet(String version) { + int versionIndex = versionIndex(version); + return versionIndex < 0 ? null : EXACT_SETS[versionIndex]; + } + + static UnicodeSet cumulativeSet(String version) { + int versionIndex = versionIndex(version); + return versionIndex < 0 ? null : CUMULATIVE_SETS[versionIndex]; + } + + static UnicodeSet unassignedSet() { + return UNASSIGNED; + } + + private static UnicodeSet[] buildExactSets() { + UnicodeSet[] sets = new UnicodeSet[VERSIONS.length]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + for (int i = 0; i < RANGES.length; i += 3) { + sets[RANGES[i + 2]].add(RANGES[i], RANGES[i + 1]); + } + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + + private static UnicodeSet[] buildCumulativeSets() { + UnicodeSet[] sets = new UnicodeSet[VERSIONS.length]; + UnicodeSet running = new UnicodeSet(); + for (int i = 0; i < sets.length; i++) { + running.addAll(EXACT_SETS[i]); + sets[i] = new UnicodeSet(running).freeze(); + } + return sets; + } + + private static int versionIndex(String version) { + for (int i = 0; i < VERSIONS.length; i++) { + if (VERSIONS[i].equals(version)) return i; + } + return -1; + } + + private PerlUnicodeAgeData() { + } +} +FOOTER diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index 3b6df3c40d..6a5999e0d5 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -371,14 +371,14 @@ my @copy = @{$z}; # ERROR - ✅ **Inline comments**: `(?#comment)` in regex is implemented. - ✅ **caret modifier**: `(?^` embedded pattern-match modifier, shorthand equivalent to "d-imnsx". - ✅ **\b inside character class**: `[\b]` is supported in regex. -- 🟡 **Unicode boundary assertions**: `\b{gcb}` and `\B{gcb}` implement GB1–GB13 and GB999 natively in Joni; all 6,285 generated GCB boundary assertions pass on both execution backends. The recognized `sb`/`lb`/`wb` forms still use simplified frontend rewrites and pass 0/231,350 authoritative generated assertions. +- 🟡 **Unicode boundary assertions**: `\b{gcb}`, `\b{sb}`, and `\b{wb}` (and their `\B` forms) execute natively in Joni with complete authoritative generated sections on both backends. Line boundaries still use a simplified frontend rewrite pending native `\b{lb}` integration. - ✅ **Variable Interpolation in Regex**: Features like `${var}` for embedding variables. - ✅ **Non-capturing groups**: `(?:...)` is implemented. - ✅ **Named Capture Groups**: Defining named capture groups using `(?...)` or `(?'name'...)` is supported. - ✅ **Backreferences to Named Groups**: Using `\k` or `\g{name}` for backreferences to named groups is supported. - ✅ **Relative Backreferences**: Using `\g{-n}` for relative backreferences. - ✅ **Basic Unicode Properties**: Common `\p{...}` and `\P{...}` forms such as `\p{L}` execute through Joni. -- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block/Age aliases execute through Joni, but the generated corpus still exposes loose property/value aliases and pinned acceptance/rejection gaps. +- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age` and cumulative `In`/`Present_In` are generated from pinned Perl 5.44 Unicode 17.0 data, including loose versions and `Unassigned`/`NA`; other generated property/value aliases still have pinned acceptance/rejection gaps. - ✅ **Possessive Quantifiers**: Quantifiers like `*+`, `++`, `?+`, and `{n,m}+`, which disable backtracking, are supported. - ✅ **Atomic Grouping**: Use of `(?>...)` for atomic groups is supported. - ✅ **`\K` assertion**: Keep left — in `s///`, text before `\K` is preserved; match variables reflect only the portion after `\K`. @@ -397,7 +397,7 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 132,578/407,367 identically on JVM and interpreter: chunks 01–04 expose property/value alias gaps, and the remaining boundary failures are line breaks. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 133,089/407,367: chunks 01–04 expose the remaining property/value alias gaps, and the remaining boundary failures are line breaks. - ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. - ✅ **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter. - ✅ **Unicode Word Boundaries**: Native `\b{wb}`/`\B{wb}` implement WB1–WB16 and WB999 from reproducible Perl 5.44 Unicode 17.0 word-break and extended-pictographic tables. Authoritative chunk 10 passes 19,510/19,510 identically on JVM and interpreter. diff --git a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java index 434938f6c6..fe471c7eff 100644 --- a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java +++ b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java @@ -190,7 +190,7 @@ private static UserPropertyTranslation translateUserDefinedProperties( boolean scriptExtensions = unnegated.matches( "(?i)^(?:scx|script[_ ]?extensions)\\s*=.*"); boolean frontendProperty = unnegated.matches( - "(?i)^(?:script|block|blk|age|in|present[_ ]?in)\\s*=.*"); + "(?i)^(?:script|block|blk|age|in|present[_ ]?in)\\s*(?:=|:(?!:)).*"); boolean perlBuiltInAlias = UnicodeResolver.isPerlBuiltInPropertyAlias(unnegated); if (frontendProperty && extendedClassBracketDepth > 0) { translated.append(pattern, i, end + 1); diff --git a/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeAgeData.java b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeAgeData.java new file mode 100644 index 0000000000..b70e08c4bb --- /dev/null +++ b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeAgeData.java @@ -0,0 +1,518 @@ +/* + * Generated from Perl 5.44's Unicode Character Database. Do not edit manually. + * + * Unicode data source copyright: + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in + * the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + */ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; + +final class PerlUnicodeAgeData { + static final String UNICODE_VERSION = "17.0.0"; + private static final String[] VERSIONS = { + "1.1", "2.0", "2.1", "3.0", "3.1", "3.2", "4.0", "4.1", "5.0", "5.1", "5.2", "6.0", "6.1", "6.2", "6.3", "7.0", "8.0", "9.0", "10.0", "11.0", "12.0", "12.1", "13.0", "14.0", "15.0", "15.1", "16.0", "17.0" + }; + + private static final int[] RANGES = { + 0x0, 0x1F5, 0, 0x1FA, 0x217, 0, 0x250, 0x2A8, 0, 0x2B0, 0x2DE, 0, + 0x2E0, 0x2E9, 0, 0x300, 0x345, 0, 0x360, 0x361, 0, 0x374, 0x375, 0, + 0x37A, 0x37A, 0, 0x37E, 0x37E, 0, 0x384, 0x38A, 0, 0x38C, 0x38C, 0, + 0x38E, 0x3A1, 0, 0x3A3, 0x3CE, 0, 0x3D0, 0x3D6, 0, 0x3DA, 0x3DA, 0, + 0x3DC, 0x3DC, 0, 0x3DE, 0x3DE, 0, 0x3E0, 0x3E0, 0, 0x3E2, 0x3F3, 0, + 0x401, 0x40C, 0, 0x40E, 0x44F, 0, 0x451, 0x45C, 0, 0x45E, 0x486, 0, + 0x490, 0x4C4, 0, 0x4C7, 0x4C8, 0, 0x4CB, 0x4CC, 0, 0x4D0, 0x4EB, 0, + 0x4EE, 0x4F5, 0, 0x4F8, 0x4F9, 0, 0x531, 0x556, 0, 0x559, 0x55F, 0, + 0x561, 0x587, 0, 0x589, 0x589, 0, 0x5B0, 0x5B9, 0, 0x5BB, 0x5C3, 0, + 0x5D0, 0x5EA, 0, 0x5F0, 0x5F4, 0, 0x60C, 0x60C, 0, 0x61B, 0x61B, 0, + 0x61F, 0x61F, 0, 0x621, 0x63A, 0, 0x640, 0x652, 0, 0x660, 0x66D, 0, + 0x670, 0x6B7, 0, 0x6BA, 0x6BE, 0, 0x6C0, 0x6CE, 0, 0x6D0, 0x6ED, 0, + 0x6F0, 0x6F9, 0, 0x901, 0x903, 0, 0x905, 0x939, 0, 0x93C, 0x94D, 0, + 0x950, 0x954, 0, 0x958, 0x970, 0, 0x981, 0x983, 0, 0x985, 0x98C, 0, + 0x98F, 0x990, 0, 0x993, 0x9A8, 0, 0x9AA, 0x9B0, 0, 0x9B2, 0x9B2, 0, + 0x9B6, 0x9B9, 0, 0x9BC, 0x9BC, 0, 0x9BE, 0x9C4, 0, 0x9C7, 0x9C8, 0, + 0x9CB, 0x9CD, 0, 0x9D7, 0x9D7, 0, 0x9DC, 0x9DD, 0, 0x9DF, 0x9E3, 0, + 0x9E6, 0x9FA, 0, 0xA02, 0xA02, 0, 0xA05, 0xA0A, 0, 0xA0F, 0xA10, 0, + 0xA13, 0xA28, 0, 0xA2A, 0xA30, 0, 0xA32, 0xA33, 0, 0xA35, 0xA36, 0, + 0xA38, 0xA39, 0, 0xA3C, 0xA3C, 0, 0xA3E, 0xA42, 0, 0xA47, 0xA48, 0, + 0xA4B, 0xA4D, 0, 0xA59, 0xA5C, 0, 0xA5E, 0xA5E, 0, 0xA66, 0xA74, 0, + 0xA81, 0xA83, 0, 0xA85, 0xA8B, 0, 0xA8D, 0xA8D, 0, 0xA8F, 0xA91, 0, + 0xA93, 0xAA8, 0, 0xAAA, 0xAB0, 0, 0xAB2, 0xAB3, 0, 0xAB5, 0xAB9, 0, + 0xABC, 0xAC5, 0, 0xAC7, 0xAC9, 0, 0xACB, 0xACD, 0, 0xAD0, 0xAD0, 0, + 0xAE0, 0xAE0, 0, 0xAE6, 0xAEF, 0, 0xB01, 0xB03, 0, 0xB05, 0xB0C, 0, + 0xB0F, 0xB10, 0, 0xB13, 0xB28, 0, 0xB2A, 0xB30, 0, 0xB32, 0xB33, 0, + 0xB36, 0xB39, 0, 0xB3C, 0xB43, 0, 0xB47, 0xB48, 0, 0xB4B, 0xB4D, 0, + 0xB56, 0xB57, 0, 0xB5C, 0xB5D, 0, 0xB5F, 0xB61, 0, 0xB66, 0xB70, 0, + 0xB82, 0xB83, 0, 0xB85, 0xB8A, 0, 0xB8E, 0xB90, 0, 0xB92, 0xB95, 0, + 0xB99, 0xB9A, 0, 0xB9C, 0xB9C, 0, 0xB9E, 0xB9F, 0, 0xBA3, 0xBA4, 0, + 0xBA8, 0xBAA, 0, 0xBAE, 0xBB5, 0, 0xBB7, 0xBB9, 0, 0xBBE, 0xBC2, 0, + 0xBC6, 0xBC8, 0, 0xBCA, 0xBCD, 0, 0xBD7, 0xBD7, 0, 0xBE7, 0xBF2, 0, + 0xC01, 0xC03, 0, 0xC05, 0xC0C, 0, 0xC0E, 0xC10, 0, 0xC12, 0xC28, 0, + 0xC2A, 0xC33, 0, 0xC35, 0xC39, 0, 0xC3E, 0xC44, 0, 0xC46, 0xC48, 0, + 0xC4A, 0xC4D, 0, 0xC55, 0xC56, 0, 0xC60, 0xC61, 0, 0xC66, 0xC6F, 0, + 0xC82, 0xC83, 0, 0xC85, 0xC8C, 0, 0xC8E, 0xC90, 0, 0xC92, 0xCA8, 0, + 0xCAA, 0xCB3, 0, 0xCB5, 0xCB9, 0, 0xCBE, 0xCC4, 0, 0xCC6, 0xCC8, 0, + 0xCCA, 0xCCD, 0, 0xCD5, 0xCD6, 0, 0xCDE, 0xCDE, 0, 0xCE0, 0xCE1, 0, + 0xCE6, 0xCEF, 0, 0xD02, 0xD03, 0, 0xD05, 0xD0C, 0, 0xD0E, 0xD10, 0, + 0xD12, 0xD28, 0, 0xD2A, 0xD39, 0, 0xD3E, 0xD43, 0, 0xD46, 0xD48, 0, + 0xD4A, 0xD4D, 0, 0xD57, 0xD57, 0, 0xD60, 0xD61, 0, 0xD66, 0xD6F, 0, + 0xE01, 0xE3A, 0, 0xE3F, 0xE5B, 0, 0xE81, 0xE82, 0, 0xE84, 0xE84, 0, + 0xE87, 0xE88, 0, 0xE8A, 0xE8A, 0, 0xE8D, 0xE8D, 0, 0xE94, 0xE97, 0, + 0xE99, 0xE9F, 0, 0xEA1, 0xEA3, 0, 0xEA5, 0xEA5, 0, 0xEA7, 0xEA7, 0, + 0xEAA, 0xEAB, 0, 0xEAD, 0xEB9, 0, 0xEBB, 0xEBD, 0, 0xEC0, 0xEC4, 0, + 0xEC6, 0xEC6, 0, 0xEC8, 0xECD, 0, 0xED0, 0xED9, 0, 0xEDC, 0xEDD, 0, + 0x10A0, 0x10C5, 0, 0x10D0, 0x10F6, 0, 0x10FB, 0x10FB, 0, 0x1100, 0x1159, 0, + 0x115F, 0x11A2, 0, 0x11A8, 0x11F9, 0, 0x1E00, 0x1E9A, 0, 0x1EA0, 0x1EF9, 0, + 0x1F00, 0x1F15, 0, 0x1F18, 0x1F1D, 0, 0x1F20, 0x1F45, 0, 0x1F48, 0x1F4D, 0, + 0x1F50, 0x1F57, 0, 0x1F59, 0x1F59, 0, 0x1F5B, 0x1F5B, 0, 0x1F5D, 0x1F5D, 0, + 0x1F5F, 0x1F7D, 0, 0x1F80, 0x1FB4, 0, 0x1FB6, 0x1FC4, 0, 0x1FC6, 0x1FD3, 0, + 0x1FD6, 0x1FDB, 0, 0x1FDD, 0x1FEF, 0, 0x1FF2, 0x1FF4, 0, 0x1FF6, 0x1FFE, 0, + 0x2000, 0x202E, 0, 0x2030, 0x2046, 0, 0x206A, 0x2070, 0, 0x2074, 0x208E, 0, + 0x20A0, 0x20AA, 0, 0x20D0, 0x20E1, 0, 0x2100, 0x2138, 0, 0x2153, 0x2182, 0, + 0x2190, 0x21EA, 0, 0x2200, 0x22F1, 0, 0x2300, 0x2300, 0, 0x2302, 0x237A, 0, + 0x2400, 0x2424, 0, 0x2440, 0x244A, 0, 0x2460, 0x24EA, 0, 0x2500, 0x2595, 0, + 0x25A0, 0x25EF, 0, 0x2600, 0x2613, 0, 0x261A, 0x266F, 0, 0x2701, 0x2704, 0, + 0x2706, 0x2709, 0, 0x270C, 0x2727, 0, 0x2729, 0x274B, 0, 0x274D, 0x274D, 0, + 0x274F, 0x2752, 0, 0x2756, 0x2756, 0, 0x2758, 0x275E, 0, 0x2761, 0x2767, 0, + 0x2776, 0x2794, 0, 0x2798, 0x27AF, 0, 0x27B1, 0x27BE, 0, 0x3000, 0x3037, 0, + 0x303F, 0x303F, 0, 0x3041, 0x3094, 0, 0x3099, 0x309E, 0, 0x30A1, 0x30FE, 0, + 0x3105, 0x312C, 0, 0x3131, 0x318E, 0, 0x3190, 0x319F, 0, 0x3200, 0x321C, 0, + 0x3220, 0x3243, 0, 0x3260, 0x327B, 0, 0x327F, 0x32B0, 0, 0x32C0, 0x32CB, 0, + 0x32D0, 0x32FE, 0, 0x3300, 0x3376, 0, 0x337B, 0x33DD, 0, 0x33E0, 0x33FE, 0, + 0x4E00, 0x9FA5, 0, 0xE000, 0xFA2D, 0, 0xFB00, 0xFB06, 0, 0xFB13, 0xFB17, 0, + 0xFB1E, 0xFB36, 0, 0xFB38, 0xFB3C, 0, 0xFB3E, 0xFB3E, 0, 0xFB40, 0xFB41, 0, + 0xFB43, 0xFB44, 0, 0xFB46, 0xFBB1, 0, 0xFBD3, 0xFD3F, 0, 0xFD50, 0xFD8F, 0, + 0xFD92, 0xFDC7, 0, 0xFDF0, 0xFDFB, 0, 0xFE20, 0xFE23, 0, 0xFE30, 0xFE44, 0, + 0xFE49, 0xFE52, 0, 0xFE54, 0xFE66, 0, 0xFE68, 0xFE6B, 0, 0xFE70, 0xFE72, 0, + 0xFE74, 0xFE74, 0, 0xFE76, 0xFEFC, 0, 0xFEFF, 0xFEFF, 0, 0xFF01, 0xFF5E, 0, + 0xFF61, 0xFFBE, 0, 0xFFC2, 0xFFC7, 0, 0xFFCA, 0xFFCF, 0, 0xFFD2, 0xFFD7, 0, + 0xFFDA, 0xFFDC, 0, 0xFFE0, 0xFFE6, 0, 0xFFE8, 0xFFEE, 0, 0xFFFD, 0xFFFF, 0, + 0x591, 0x5A1, 1, 0x5A3, 0x5AF, 1, 0x5C4, 0x5C4, 1, 0xF00, 0xF47, 1, + 0xF49, 0xF69, 1, 0xF71, 0xF8B, 1, 0xF90, 0xF95, 1, 0xF97, 0xF97, 1, + 0xF99, 0xFAD, 1, 0xFB1, 0xFB7, 1, 0xFB9, 0xFB9, 1, 0x1E9B, 0x1E9B, 1, + 0x20AB, 0x20AB, 1, 0xAC00, 0xD7A3, 1, 0xD800, 0xDFFF, 1, 0x1FFFE, 0x1FFFF, 1, + 0x2FFFE, 0x2FFFF, 1, 0x3FFFE, 0x3FFFF, 1, 0x4FFFE, 0x4FFFF, 1, 0x5FFFE, 0x5FFFF, 1, + 0x6FFFE, 0x6FFFF, 1, 0x7FFFE, 0x7FFFF, 1, 0x8FFFE, 0x8FFFF, 1, 0x9FFFE, 0x9FFFF, 1, + 0xAFFFE, 0xAFFFF, 1, 0xBFFFE, 0xBFFFF, 1, 0xCFFFE, 0xCFFFF, 1, 0xDFFFE, 0xDFFFF, 1, + 0xEFFFE, 0x10FFFF, 1, 0x20AC, 0x20AC, 2, 0xFFFC, 0xFFFC, 2, 0x1F6, 0x1F9, 3, + 0x218, 0x21F, 3, 0x222, 0x233, 3, 0x2A9, 0x2AD, 3, 0x2DF, 0x2DF, 3, + 0x2EA, 0x2EE, 3, 0x346, 0x34E, 3, 0x362, 0x362, 3, 0x3D7, 0x3D7, 3, + 0x3DB, 0x3DB, 3, 0x3DD, 0x3DD, 3, 0x3DF, 0x3DF, 3, 0x3E1, 0x3E1, 3, + 0x400, 0x400, 3, 0x40D, 0x40D, 3, 0x450, 0x450, 3, 0x45D, 0x45D, 3, + 0x488, 0x489, 3, 0x48C, 0x48F, 3, 0x4EC, 0x4ED, 3, 0x58A, 0x58A, 3, + 0x653, 0x655, 3, 0x6B8, 0x6B9, 3, 0x6BF, 0x6BF, 3, 0x6CF, 0x6CF, 3, + 0x6FA, 0x6FE, 3, 0x700, 0x70D, 3, 0x70F, 0x72C, 3, 0x730, 0x74A, 3, + 0x780, 0x7B0, 3, 0xD82, 0xD83, 3, 0xD85, 0xD96, 3, 0xD9A, 0xDB1, 3, + 0xDB3, 0xDBB, 3, 0xDBD, 0xDBD, 3, 0xDC0, 0xDC6, 3, 0xDCA, 0xDCA, 3, + 0xDCF, 0xDD4, 3, 0xDD6, 0xDD6, 3, 0xDD8, 0xDDF, 3, 0xDF2, 0xDF4, 3, + 0xF6A, 0xF6A, 3, 0xF96, 0xF96, 3, 0xFAE, 0xFB0, 3, 0xFB8, 0xFB8, 3, + 0xFBA, 0xFBC, 3, 0xFBE, 0xFCC, 3, 0xFCF, 0xFCF, 3, 0x1000, 0x1021, 3, + 0x1023, 0x1027, 3, 0x1029, 0x102A, 3, 0x102C, 0x1032, 3, 0x1036, 0x1039, 3, + 0x1040, 0x1059, 3, 0x1200, 0x1206, 3, 0x1208, 0x1246, 3, 0x1248, 0x1248, 3, + 0x124A, 0x124D, 3, 0x1250, 0x1256, 3, 0x1258, 0x1258, 3, 0x125A, 0x125D, 3, + 0x1260, 0x1286, 3, 0x1288, 0x1288, 3, 0x128A, 0x128D, 3, 0x1290, 0x12AE, 3, + 0x12B0, 0x12B0, 3, 0x12B2, 0x12B5, 3, 0x12B8, 0x12BE, 3, 0x12C0, 0x12C0, 3, + 0x12C2, 0x12C5, 3, 0x12C8, 0x12CE, 3, 0x12D0, 0x12D6, 3, 0x12D8, 0x12EE, 3, + 0x12F0, 0x130E, 3, 0x1310, 0x1310, 3, 0x1312, 0x1315, 3, 0x1318, 0x131E, 3, + 0x1320, 0x1346, 3, 0x1348, 0x135A, 3, 0x1361, 0x137C, 3, 0x13A0, 0x13F4, 3, + 0x1401, 0x1676, 3, 0x1680, 0x169C, 3, 0x16A0, 0x16F0, 3, 0x1780, 0x17DC, 3, + 0x17E0, 0x17E9, 3, 0x1800, 0x180E, 3, 0x1810, 0x1819, 3, 0x1820, 0x1877, 3, + 0x1880, 0x18A9, 3, 0x202F, 0x202F, 3, 0x2048, 0x204D, 3, 0x20AD, 0x20AF, 3, + 0x20E2, 0x20E3, 3, 0x2139, 0x213A, 3, 0x2183, 0x2183, 3, 0x21EB, 0x21F3, 3, + 0x2301, 0x2301, 3, 0x237B, 0x237B, 3, 0x237D, 0x239A, 3, 0x2425, 0x2426, 3, + 0x25F0, 0x25F7, 3, 0x2619, 0x2619, 3, 0x2670, 0x2671, 3, 0x2800, 0x28FF, 3, + 0x2E80, 0x2E99, 3, 0x2E9B, 0x2EF3, 3, 0x2F00, 0x2FD5, 3, 0x2FF0, 0x2FFB, 3, + 0x3038, 0x303A, 3, 0x303E, 0x303E, 3, 0x31A0, 0x31B7, 3, 0x3400, 0x4DB5, 3, + 0xA000, 0xA48C, 3, 0xA490, 0xA4A1, 3, 0xA4A4, 0xA4B3, 3, 0xA4B5, 0xA4C0, 3, + 0xA4C2, 0xA4C4, 3, 0xA4C6, 0xA4C6, 3, 0xFB1D, 0xFB1D, 3, 0xFFF9, 0xFFFB, 3, + 0x3F4, 0x3F5, 4, 0xFDD0, 0xFDEF, 4, 0x10300, 0x1031E, 4, 0x10320, 0x10323, 4, + 0x10330, 0x1034A, 4, 0x10400, 0x10425, 4, 0x10428, 0x1044D, 4, 0x1D000, 0x1D0F5, 4, + 0x1D100, 0x1D126, 4, 0x1D12A, 0x1D1DD, 4, 0x1D400, 0x1D454, 4, 0x1D456, 0x1D49C, 4, + 0x1D49E, 0x1D49F, 4, 0x1D4A2, 0x1D4A2, 4, 0x1D4A5, 0x1D4A6, 4, 0x1D4A9, 0x1D4AC, 4, + 0x1D4AE, 0x1D4B9, 4, 0x1D4BB, 0x1D4BB, 4, 0x1D4BD, 0x1D4C0, 4, 0x1D4C2, 0x1D4C3, 4, + 0x1D4C5, 0x1D505, 4, 0x1D507, 0x1D50A, 4, 0x1D50D, 0x1D514, 4, 0x1D516, 0x1D51C, 4, + 0x1D51E, 0x1D539, 4, 0x1D53B, 0x1D53E, 4, 0x1D540, 0x1D544, 4, 0x1D546, 0x1D546, 4, + 0x1D54A, 0x1D550, 4, 0x1D552, 0x1D6A3, 4, 0x1D6A8, 0x1D7C9, 4, 0x1D7CE, 0x1D7FF, 4, + 0x20000, 0x2A6D6, 4, 0x2F800, 0x2FA1D, 4, 0xE0001, 0xE0001, 4, 0xE0020, 0xE007F, 4, + 0x220, 0x220, 5, 0x34F, 0x34F, 5, 0x363, 0x36F, 5, 0x3D8, 0x3D9, 5, + 0x3F6, 0x3F6, 5, 0x48A, 0x48B, 5, 0x4C5, 0x4C6, 5, 0x4C9, 0x4CA, 5, + 0x4CD, 0x4CE, 5, 0x500, 0x50F, 5, 0x66E, 0x66F, 5, 0x7B1, 0x7B1, 5, + 0x10F7, 0x10F8, 5, 0x1700, 0x170C, 5, 0x170E, 0x1714, 5, 0x1720, 0x1736, 5, + 0x1740, 0x1753, 5, 0x1760, 0x176C, 5, 0x176E, 0x1770, 5, 0x1772, 0x1773, 5, + 0x2047, 0x2047, 5, 0x204E, 0x2052, 5, 0x2057, 0x2057, 5, 0x205F, 0x2063, 5, + 0x2071, 0x2071, 5, 0x20B0, 0x20B1, 5, 0x20E4, 0x20EA, 5, 0x213D, 0x214B, 5, + 0x21F4, 0x21FF, 5, 0x22F2, 0x22FF, 5, 0x237C, 0x237C, 5, 0x239B, 0x23CE, 5, + 0x24EB, 0x24FE, 5, 0x2596, 0x259F, 5, 0x25F8, 0x25FF, 5, 0x2616, 0x2617, 5, + 0x2672, 0x267D, 5, 0x2680, 0x2689, 5, 0x2768, 0x2775, 5, 0x27D0, 0x27EB, 5, + 0x27F0, 0x27FF, 5, 0x2900, 0x2AFF, 5, 0x303B, 0x303D, 5, 0x3095, 0x3096, 5, + 0x309F, 0x30A0, 5, 0x30FF, 0x30FF, 5, 0x31F0, 0x31FF, 5, 0x3251, 0x325F, 5, + 0x32B1, 0x32BF, 5, 0xA4A2, 0xA4A3, 5, 0xA4B4, 0xA4B4, 5, 0xA4C1, 0xA4C1, 5, + 0xA4C5, 0xA4C5, 5, 0xFA30, 0xFA6A, 5, 0xFDFC, 0xFDFC, 5, 0xFE00, 0xFE0F, 5, + 0xFE45, 0xFE46, 5, 0xFE73, 0xFE73, 5, 0xFF5F, 0xFF60, 5, 0x221, 0x221, 6, + 0x234, 0x236, 6, 0x2AE, 0x2AF, 6, 0x2EF, 0x2FF, 6, 0x350, 0x357, 6, + 0x35D, 0x35F, 6, 0x3F7, 0x3FB, 6, 0x600, 0x603, 6, 0x60D, 0x615, 6, + 0x656, 0x658, 6, 0x6EE, 0x6EF, 6, 0x6FF, 0x6FF, 6, 0x72D, 0x72F, 6, + 0x74D, 0x74F, 6, 0x904, 0x904, 6, 0x9BD, 0x9BD, 6, 0xA01, 0xA01, 6, + 0xA03, 0xA03, 6, 0xA8C, 0xA8C, 6, 0xAE1, 0xAE3, 6, 0xAF1, 0xAF1, 6, + 0xB35, 0xB35, 6, 0xB71, 0xB71, 6, 0xBF3, 0xBFA, 6, 0xCBC, 0xCBD, 6, + 0x17DD, 0x17DD, 6, 0x17F0, 0x17F9, 6, 0x1900, 0x191C, 6, 0x1920, 0x192B, 6, + 0x1930, 0x193B, 6, 0x1940, 0x1940, 6, 0x1944, 0x196D, 6, 0x1970, 0x1974, 6, + 0x19E0, 0x19FF, 6, 0x1D00, 0x1D6B, 6, 0x2053, 0x2054, 6, 0x213B, 0x213B, 6, + 0x23CF, 0x23D0, 6, 0x24FF, 0x24FF, 6, 0x2614, 0x2615, 6, 0x268A, 0x2691, 6, + 0x26A0, 0x26A1, 6, 0x2B00, 0x2B0D, 6, 0x321D, 0x321E, 6, 0x3250, 0x3250, 6, + 0x327C, 0x327D, 6, 0x32CC, 0x32CF, 6, 0x3377, 0x337A, 6, 0x33DE, 0x33DF, 6, + 0x33FF, 0x33FF, 6, 0x4DC0, 0x4DFF, 6, 0xFDFD, 0xFDFD, 6, 0xFE47, 0xFE48, 6, + 0x10000, 0x1000B, 6, 0x1000D, 0x10026, 6, 0x10028, 0x1003A, 6, 0x1003C, 0x1003D, 6, + 0x1003F, 0x1004D, 6, 0x10050, 0x1005D, 6, 0x10080, 0x100FA, 6, 0x10100, 0x10102, 6, + 0x10107, 0x10133, 6, 0x10137, 0x1013F, 6, 0x10380, 0x1039D, 6, 0x1039F, 0x1039F, 6, + 0x10426, 0x10427, 6, 0x1044E, 0x1049D, 6, 0x104A0, 0x104A9, 6, 0x10800, 0x10805, 6, + 0x10808, 0x10808, 6, 0x1080A, 0x10835, 6, 0x10837, 0x10838, 6, 0x1083C, 0x1083C, 6, + 0x1083F, 0x1083F, 6, 0x1D300, 0x1D356, 6, 0x1D4C1, 0x1D4C1, 6, 0xE0100, 0xE01EF, 6, + 0x237, 0x241, 7, 0x358, 0x35C, 7, 0x3FC, 0x3FF, 7, 0x4F6, 0x4F7, 7, + 0x5A2, 0x5A2, 7, 0x5C5, 0x5C7, 7, 0x60B, 0x60B, 7, 0x61E, 0x61E, 7, + 0x659, 0x65E, 7, 0x750, 0x76D, 7, 0x97D, 0x97D, 7, 0x9CE, 0x9CE, 7, + 0xBB6, 0xBB6, 7, 0xBE6, 0xBE6, 7, 0xFD0, 0xFD1, 7, 0x10F9, 0x10FA, 7, + 0x10FC, 0x10FC, 7, 0x1207, 0x1207, 7, 0x1247, 0x1247, 7, 0x1287, 0x1287, 7, + 0x12AF, 0x12AF, 7, 0x12CF, 0x12CF, 7, 0x12EF, 0x12EF, 7, 0x130F, 0x130F, 7, + 0x131F, 0x131F, 7, 0x1347, 0x1347, 7, 0x135F, 0x1360, 7, 0x1380, 0x1399, 7, + 0x1980, 0x19A9, 7, 0x19B0, 0x19C9, 7, 0x19D0, 0x19D9, 7, 0x19DE, 0x19DF, 7, + 0x1A00, 0x1A1B, 7, 0x1A1E, 0x1A1F, 7, 0x1D6C, 0x1DC3, 7, 0x2055, 0x2056, 7, + 0x2058, 0x205E, 7, 0x2090, 0x2094, 7, 0x20B2, 0x20B5, 7, 0x20EB, 0x20EB, 7, + 0x213C, 0x213C, 7, 0x214C, 0x214C, 7, 0x23D1, 0x23DB, 7, 0x2618, 0x2618, 7, + 0x267E, 0x267F, 7, 0x2692, 0x269C, 7, 0x26A2, 0x26B1, 7, 0x27C0, 0x27C6, 7, + 0x2B0E, 0x2B13, 7, 0x2C00, 0x2C2E, 7, 0x2C30, 0x2C5E, 7, 0x2C80, 0x2CEA, 7, + 0x2CF9, 0x2D25, 7, 0x2D30, 0x2D65, 7, 0x2D6F, 0x2D6F, 7, 0x2D80, 0x2D96, 7, + 0x2DA0, 0x2DA6, 7, 0x2DA8, 0x2DAE, 7, 0x2DB0, 0x2DB6, 7, 0x2DB8, 0x2DBE, 7, + 0x2DC0, 0x2DC6, 7, 0x2DC8, 0x2DCE, 7, 0x2DD0, 0x2DD6, 7, 0x2DD8, 0x2DDE, 7, + 0x2E00, 0x2E17, 7, 0x2E1C, 0x2E1D, 7, 0x31C0, 0x31CF, 7, 0x327E, 0x327E, 7, + 0x9FA6, 0x9FBB, 7, 0xA700, 0xA716, 7, 0xA800, 0xA82B, 7, 0xFA70, 0xFAD9, 7, + 0xFE10, 0xFE19, 7, 0x10140, 0x1018A, 7, 0x103A0, 0x103C3, 7, 0x103C8, 0x103D5, 7, + 0x10A00, 0x10A03, 7, 0x10A05, 0x10A06, 7, 0x10A0C, 0x10A13, 7, 0x10A15, 0x10A17, 7, + 0x10A19, 0x10A33, 7, 0x10A38, 0x10A3A, 7, 0x10A3F, 0x10A47, 7, 0x10A50, 0x10A58, 7, + 0x1D200, 0x1D245, 7, 0x1D6A4, 0x1D6A5, 7, 0x242, 0x24F, 8, 0x37B, 0x37D, 8, + 0x4CF, 0x4CF, 8, 0x4FA, 0x4FF, 8, 0x510, 0x513, 8, 0x5BA, 0x5BA, 8, + 0x7C0, 0x7FA, 8, 0x97B, 0x97C, 8, 0x97E, 0x97F, 8, 0xCE2, 0xCE3, 8, + 0xCF1, 0xCF2, 8, 0x1B00, 0x1B4B, 8, 0x1B50, 0x1B7C, 8, 0x1DC4, 0x1DCA, 8, + 0x1DFE, 0x1DFF, 8, 0x20EC, 0x20EF, 8, 0x214D, 0x214E, 8, 0x2184, 0x2184, 8, + 0x23DC, 0x23E7, 8, 0x26B2, 0x26B2, 8, 0x27C7, 0x27CA, 8, 0x2B14, 0x2B1A, 8, + 0x2B20, 0x2B23, 8, 0x2C60, 0x2C6C, 8, 0x2C74, 0x2C77, 8, 0xA717, 0xA71A, 8, + 0xA720, 0xA721, 8, 0xA840, 0xA877, 8, 0x10900, 0x10919, 8, 0x1091F, 0x1091F, 8, + 0x12000, 0x1236E, 8, 0x12400, 0x12462, 8, 0x12470, 0x12473, 8, 0x1D360, 0x1D371, 8, + 0x1D7CA, 0x1D7CB, 8, 0x370, 0x373, 9, 0x376, 0x377, 9, 0x3CF, 0x3CF, 9, + 0x487, 0x487, 9, 0x514, 0x523, 9, 0x606, 0x60A, 9, 0x616, 0x61A, 9, + 0x63B, 0x63F, 9, 0x76E, 0x77F, 9, 0x971, 0x972, 9, 0xA51, 0xA51, 9, + 0xA75, 0xA75, 9, 0xB44, 0xB44, 9, 0xB62, 0xB63, 9, 0xBD0, 0xBD0, 9, + 0xC3D, 0xC3D, 9, 0xC58, 0xC59, 9, 0xC62, 0xC63, 9, 0xC78, 0xC7F, 9, + 0xD3D, 0xD3D, 9, 0xD44, 0xD44, 9, 0xD62, 0xD63, 9, 0xD70, 0xD75, 9, + 0xD79, 0xD7F, 9, 0xF6B, 0xF6C, 9, 0xFCE, 0xFCE, 9, 0xFD2, 0xFD4, 9, + 0x1022, 0x1022, 9, 0x1028, 0x1028, 9, 0x102B, 0x102B, 9, 0x1033, 0x1035, 9, + 0x103A, 0x103F, 9, 0x105A, 0x1099, 9, 0x109E, 0x109F, 9, 0x18AA, 0x18AA, 9, + 0x1B80, 0x1BAA, 9, 0x1BAE, 0x1BB9, 9, 0x1C00, 0x1C37, 9, 0x1C3B, 0x1C49, 9, + 0x1C4D, 0x1C7F, 9, 0x1DCB, 0x1DE6, 9, 0x1E9C, 0x1E9F, 9, 0x1EFA, 0x1EFF, 9, + 0x2064, 0x2064, 9, 0x20F0, 0x20F0, 9, 0x214F, 0x214F, 9, 0x2185, 0x2188, 9, + 0x269D, 0x269D, 9, 0x26B3, 0x26BC, 9, 0x26C0, 0x26C3, 9, 0x27CC, 0x27CC, 9, + 0x27EC, 0x27EF, 9, 0x2B1B, 0x2B1F, 9, 0x2B24, 0x2B4C, 9, 0x2B50, 0x2B54, 9, + 0x2C6D, 0x2C6F, 9, 0x2C71, 0x2C73, 9, 0x2C78, 0x2C7D, 9, 0x2DE0, 0x2DFF, 9, + 0x2E18, 0x2E1B, 9, 0x2E1E, 0x2E30, 9, 0x312D, 0x312D, 9, 0x31D0, 0x31E3, 9, + 0x9FBC, 0x9FC3, 9, 0xA500, 0xA62B, 9, 0xA640, 0xA65F, 9, 0xA662, 0xA673, 9, + 0xA67C, 0xA697, 9, 0xA71B, 0xA71F, 9, 0xA722, 0xA78C, 9, 0xA7FB, 0xA7FF, 9, + 0xA880, 0xA8C4, 9, 0xA8CE, 0xA8D9, 9, 0xA900, 0xA953, 9, 0xA95F, 0xA95F, 9, + 0xAA00, 0xAA36, 9, 0xAA40, 0xAA4D, 9, 0xAA50, 0xAA59, 9, 0xAA5C, 0xAA5F, 9, + 0xFE24, 0xFE26, 9, 0x10190, 0x1019B, 9, 0x101D0, 0x101FD, 9, 0x10280, 0x1029C, 9, + 0x102A0, 0x102D0, 9, 0x10920, 0x10939, 9, 0x1093F, 0x1093F, 9, 0x1D129, 0x1D129, 9, + 0x1F000, 0x1F02B, 9, 0x1F030, 0x1F093, 9, 0x524, 0x525, 10, 0x800, 0x82D, 10, + 0x830, 0x83E, 10, 0x900, 0x900, 10, 0x94E, 0x94E, 10, 0x955, 0x955, 10, + 0x979, 0x97A, 10, 0x9FB, 0x9FB, 10, 0xFD5, 0xFD8, 10, 0x109A, 0x109D, 10, + 0x115A, 0x115E, 10, 0x11A3, 0x11A7, 10, 0x11FA, 0x11FF, 10, 0x1400, 0x1400, 10, + 0x1677, 0x167F, 10, 0x18B0, 0x18F5, 10, 0x19AA, 0x19AB, 10, 0x19DA, 0x19DA, 10, + 0x1A20, 0x1A5E, 10, 0x1A60, 0x1A7C, 10, 0x1A7F, 0x1A89, 10, 0x1A90, 0x1A99, 10, + 0x1AA0, 0x1AAD, 10, 0x1CD0, 0x1CF2, 10, 0x1DFD, 0x1DFD, 10, 0x20B6, 0x20B8, 10, + 0x2150, 0x2152, 10, 0x2189, 0x2189, 10, 0x23E8, 0x23E8, 10, 0x269E, 0x269F, 10, + 0x26BD, 0x26BF, 10, 0x26C4, 0x26CD, 10, 0x26CF, 0x26E1, 10, 0x26E3, 0x26E3, 10, + 0x26E8, 0x26FF, 10, 0x2757, 0x2757, 10, 0x2B55, 0x2B59, 10, 0x2C70, 0x2C70, 10, + 0x2C7E, 0x2C7F, 10, 0x2CEB, 0x2CF1, 10, 0x2E31, 0x2E31, 10, 0x3244, 0x324F, 10, + 0x9FC4, 0x9FCB, 10, 0xA4D0, 0xA4FF, 10, 0xA6A0, 0xA6F7, 10, 0xA830, 0xA839, 10, + 0xA8E0, 0xA8FB, 10, 0xA960, 0xA97C, 10, 0xA980, 0xA9CD, 10, 0xA9CF, 0xA9D9, 10, + 0xA9DE, 0xA9DF, 10, 0xAA60, 0xAA7B, 10, 0xAA80, 0xAAC2, 10, 0xAADB, 0xAADF, 10, + 0xABC0, 0xABED, 10, 0xABF0, 0xABF9, 10, 0xD7B0, 0xD7C6, 10, 0xD7CB, 0xD7FB, 10, + 0xFA6B, 0xFA6D, 10, 0x10840, 0x10855, 10, 0x10857, 0x1085F, 10, 0x1091A, 0x1091B, 10, + 0x10A60, 0x10A7F, 10, 0x10B00, 0x10B35, 10, 0x10B39, 0x10B55, 10, 0x10B58, 0x10B72, 10, + 0x10B78, 0x10B7F, 10, 0x10C00, 0x10C48, 10, 0x10E60, 0x10E7E, 10, 0x11080, 0x110C1, 10, + 0x13000, 0x1342E, 10, 0x1F100, 0x1F10A, 10, 0x1F110, 0x1F12E, 10, 0x1F131, 0x1F131, 10, + 0x1F13D, 0x1F13D, 10, 0x1F13F, 0x1F13F, 10, 0x1F142, 0x1F142, 10, 0x1F146, 0x1F146, 10, + 0x1F14A, 0x1F14E, 10, 0x1F157, 0x1F157, 10, 0x1F15F, 0x1F15F, 10, 0x1F179, 0x1F179, 10, + 0x1F17B, 0x1F17C, 10, 0x1F17F, 0x1F17F, 10, 0x1F18A, 0x1F18D, 10, 0x1F190, 0x1F190, 10, + 0x1F200, 0x1F200, 10, 0x1F210, 0x1F231, 10, 0x1F240, 0x1F248, 10, 0x2A700, 0x2B734, 10, + 0x526, 0x527, 11, 0x620, 0x620, 11, 0x65F, 0x65F, 11, 0x840, 0x85B, 11, + 0x85E, 0x85E, 11, 0x93A, 0x93B, 11, 0x94F, 0x94F, 11, 0x956, 0x957, 11, + 0x973, 0x977, 11, 0xB72, 0xB77, 11, 0xD29, 0xD29, 11, 0xD3A, 0xD3A, 11, + 0xD4E, 0xD4E, 11, 0xF8C, 0xF8F, 11, 0xFD9, 0xFDA, 11, 0x135D, 0x135E, 11, + 0x1BC0, 0x1BF3, 11, 0x1BFC, 0x1BFF, 11, 0x1DFC, 0x1DFC, 11, 0x2095, 0x209C, 11, + 0x20B9, 0x20B9, 11, 0x23E9, 0x23F3, 11, 0x26CE, 0x26CE, 11, 0x26E2, 0x26E2, 11, + 0x26E4, 0x26E7, 11, 0x2705, 0x2705, 11, 0x270A, 0x270B, 11, 0x2728, 0x2728, 11, + 0x274C, 0x274C, 11, 0x274E, 0x274E, 11, 0x2753, 0x2755, 11, 0x275F, 0x2760, 11, + 0x2795, 0x2797, 11, 0x27B0, 0x27B0, 11, 0x27BF, 0x27BF, 11, 0x27CE, 0x27CF, 11, + 0x2D70, 0x2D70, 11, 0x2D7F, 0x2D7F, 11, 0x31B8, 0x31BA, 11, 0xA660, 0xA661, 11, + 0xA78D, 0xA78E, 11, 0xA790, 0xA791, 11, 0xA7A0, 0xA7A9, 11, 0xA7FA, 0xA7FA, 11, + 0xAB01, 0xAB06, 11, 0xAB09, 0xAB0E, 11, 0xAB11, 0xAB16, 11, 0xAB20, 0xAB26, 11, + 0xAB28, 0xAB2E, 11, 0xFBB2, 0xFBC1, 11, 0x11000, 0x1104D, 11, 0x11052, 0x1106F, 11, + 0x16800, 0x16A38, 11, 0x1B000, 0x1B001, 11, 0x1F0A0, 0x1F0AE, 11, 0x1F0B1, 0x1F0BE, 11, + 0x1F0C1, 0x1F0CF, 11, 0x1F0D1, 0x1F0DF, 11, 0x1F130, 0x1F130, 11, 0x1F132, 0x1F13C, 11, + 0x1F13E, 0x1F13E, 11, 0x1F140, 0x1F141, 11, 0x1F143, 0x1F145, 11, 0x1F147, 0x1F149, 11, + 0x1F14F, 0x1F156, 11, 0x1F158, 0x1F15E, 11, 0x1F160, 0x1F169, 11, 0x1F170, 0x1F178, 11, + 0x1F17A, 0x1F17A, 11, 0x1F17D, 0x1F17E, 11, 0x1F180, 0x1F189, 11, 0x1F18E, 0x1F18F, 11, + 0x1F191, 0x1F19A, 11, 0x1F1E6, 0x1F1FF, 11, 0x1F201, 0x1F202, 11, 0x1F232, 0x1F23A, 11, + 0x1F250, 0x1F251, 11, 0x1F300, 0x1F320, 11, 0x1F330, 0x1F335, 11, 0x1F337, 0x1F37C, 11, + 0x1F380, 0x1F393, 11, 0x1F3A0, 0x1F3C4, 11, 0x1F3C6, 0x1F3CA, 11, 0x1F3E0, 0x1F3F0, 11, + 0x1F400, 0x1F43E, 11, 0x1F440, 0x1F440, 11, 0x1F442, 0x1F4F7, 11, 0x1F4F9, 0x1F4FC, 11, + 0x1F500, 0x1F53D, 11, 0x1F550, 0x1F567, 11, 0x1F5FB, 0x1F5FF, 11, 0x1F601, 0x1F610, 11, + 0x1F612, 0x1F614, 11, 0x1F616, 0x1F616, 11, 0x1F618, 0x1F618, 11, 0x1F61A, 0x1F61A, 11, + 0x1F61C, 0x1F61E, 11, 0x1F620, 0x1F625, 11, 0x1F628, 0x1F62B, 11, 0x1F62D, 0x1F62D, 11, + 0x1F630, 0x1F633, 11, 0x1F635, 0x1F640, 11, 0x1F645, 0x1F64F, 11, 0x1F680, 0x1F6C5, 11, + 0x1F700, 0x1F773, 11, 0x2B740, 0x2B81D, 11, 0x58F, 0x58F, 12, 0x604, 0x604, 12, + 0x8A0, 0x8A0, 12, 0x8A2, 0x8AC, 12, 0x8E4, 0x8FE, 12, 0xAF0, 0xAF0, 12, + 0xEDE, 0xEDF, 12, 0x10C7, 0x10C7, 12, 0x10CD, 0x10CD, 12, 0x10FD, 0x10FF, 12, + 0x1BAB, 0x1BAD, 12, 0x1BBA, 0x1BBF, 12, 0x1CC0, 0x1CC7, 12, 0x1CF3, 0x1CF6, 12, + 0x27CB, 0x27CB, 12, 0x27CD, 0x27CD, 12, 0x2CF2, 0x2CF3, 12, 0x2D27, 0x2D27, 12, + 0x2D2D, 0x2D2D, 12, 0x2D66, 0x2D67, 12, 0x2E32, 0x2E3B, 12, 0x9FCC, 0x9FCC, 12, + 0xA674, 0xA67B, 12, 0xA69F, 0xA69F, 12, 0xA792, 0xA793, 12, 0xA7AA, 0xA7AA, 12, + 0xA7F8, 0xA7F9, 12, 0xAAE0, 0xAAF6, 12, 0xFA2E, 0xFA2F, 12, 0x10980, 0x109B7, 12, + 0x109BE, 0x109BF, 12, 0x110D0, 0x110E8, 12, 0x110F0, 0x110F9, 12, 0x11100, 0x11134, 12, + 0x11136, 0x11143, 12, 0x11180, 0x111C8, 12, 0x111D0, 0x111D9, 12, 0x11680, 0x116B7, 12, + 0x116C0, 0x116C9, 12, 0x16F00, 0x16F44, 12, 0x16F50, 0x16F7E, 12, 0x16F8F, 0x16F9F, 12, + 0x1EE00, 0x1EE03, 12, 0x1EE05, 0x1EE1F, 12, 0x1EE21, 0x1EE22, 12, 0x1EE24, 0x1EE24, 12, + 0x1EE27, 0x1EE27, 12, 0x1EE29, 0x1EE32, 12, 0x1EE34, 0x1EE37, 12, 0x1EE39, 0x1EE39, 12, + 0x1EE3B, 0x1EE3B, 12, 0x1EE42, 0x1EE42, 12, 0x1EE47, 0x1EE47, 12, 0x1EE49, 0x1EE49, 12, + 0x1EE4B, 0x1EE4B, 12, 0x1EE4D, 0x1EE4F, 12, 0x1EE51, 0x1EE52, 12, 0x1EE54, 0x1EE54, 12, + 0x1EE57, 0x1EE57, 12, 0x1EE59, 0x1EE59, 12, 0x1EE5B, 0x1EE5B, 12, 0x1EE5D, 0x1EE5D, 12, + 0x1EE5F, 0x1EE5F, 12, 0x1EE61, 0x1EE62, 12, 0x1EE64, 0x1EE64, 12, 0x1EE67, 0x1EE6A, 12, + 0x1EE6C, 0x1EE72, 12, 0x1EE74, 0x1EE77, 12, 0x1EE79, 0x1EE7C, 12, 0x1EE7E, 0x1EE7E, 12, + 0x1EE80, 0x1EE89, 12, 0x1EE8B, 0x1EE9B, 12, 0x1EEA1, 0x1EEA3, 12, 0x1EEA5, 0x1EEA9, 12, + 0x1EEAB, 0x1EEBB, 12, 0x1EEF0, 0x1EEF1, 12, 0x1F16A, 0x1F16B, 12, 0x1F540, 0x1F543, 12, + 0x1F600, 0x1F600, 12, 0x1F611, 0x1F611, 12, 0x1F615, 0x1F615, 12, 0x1F617, 0x1F617, 12, + 0x1F619, 0x1F619, 12, 0x1F61B, 0x1F61B, 12, 0x1F61F, 0x1F61F, 12, 0x1F626, 0x1F627, 12, + 0x1F62C, 0x1F62C, 12, 0x1F62E, 0x1F62F, 12, 0x1F634, 0x1F634, 12, 0x20BA, 0x20BA, 13, + 0x61C, 0x61C, 14, 0x2066, 0x2069, 14, 0x37F, 0x37F, 15, 0x528, 0x52F, 15, + 0x58D, 0x58E, 15, 0x605, 0x605, 15, 0x8A1, 0x8A1, 15, 0x8AD, 0x8B2, 15, + 0x8FF, 0x8FF, 15, 0x978, 0x978, 15, 0x980, 0x980, 15, 0xC00, 0xC00, 15, + 0xC34, 0xC34, 15, 0xC81, 0xC81, 15, 0xD01, 0xD01, 15, 0xDE6, 0xDEF, 15, + 0x16F1, 0x16F8, 15, 0x191D, 0x191E, 15, 0x1AB0, 0x1ABE, 15, 0x1CF8, 0x1CF9, 15, + 0x1DE7, 0x1DF5, 15, 0x20BB, 0x20BD, 15, 0x23F4, 0x23FA, 15, 0x2700, 0x2700, 15, + 0x2B4D, 0x2B4F, 15, 0x2B5A, 0x2B73, 15, 0x2B76, 0x2B95, 15, 0x2B98, 0x2BB9, 15, + 0x2BBD, 0x2BC8, 15, 0x2BCA, 0x2BD1, 15, 0x2E3C, 0x2E42, 15, 0xA698, 0xA69D, 15, + 0xA794, 0xA79F, 15, 0xA7AB, 0xA7AD, 15, 0xA7B0, 0xA7B1, 15, 0xA7F7, 0xA7F7, 15, + 0xA9E0, 0xA9FE, 15, 0xAA7C, 0xAA7F, 15, 0xAB30, 0xAB5F, 15, 0xAB64, 0xAB65, 15, + 0xFE27, 0xFE2D, 15, 0x1018B, 0x1018C, 15, 0x101A0, 0x101A0, 15, 0x102E0, 0x102FB, 15, + 0x1031F, 0x1031F, 15, 0x10350, 0x1037A, 15, 0x10500, 0x10527, 15, 0x10530, 0x10563, 15, + 0x1056F, 0x1056F, 15, 0x10600, 0x10736, 15, 0x10740, 0x10755, 15, 0x10760, 0x10767, 15, + 0x10860, 0x1089E, 15, 0x108A7, 0x108AF, 15, 0x10A80, 0x10A9F, 15, 0x10AC0, 0x10AE6, 15, + 0x10AEB, 0x10AF6, 15, 0x10B80, 0x10B91, 15, 0x10B99, 0x10B9C, 15, 0x10BA9, 0x10BAF, 15, + 0x1107F, 0x1107F, 15, 0x11150, 0x11176, 15, 0x111CD, 0x111CD, 15, 0x111DA, 0x111DA, 15, + 0x111E1, 0x111F4, 15, 0x11200, 0x11211, 15, 0x11213, 0x1123D, 15, 0x112B0, 0x112EA, 15, + 0x112F0, 0x112F9, 15, 0x11301, 0x11303, 15, 0x11305, 0x1130C, 15, 0x1130F, 0x11310, 15, + 0x11313, 0x11328, 15, 0x1132A, 0x11330, 15, 0x11332, 0x11333, 15, 0x11335, 0x11339, 15, + 0x1133C, 0x11344, 15, 0x11347, 0x11348, 15, 0x1134B, 0x1134D, 15, 0x11357, 0x11357, 15, + 0x1135D, 0x11363, 15, 0x11366, 0x1136C, 15, 0x11370, 0x11374, 15, 0x11480, 0x114C7, 15, + 0x114D0, 0x114D9, 15, 0x11580, 0x115B5, 15, 0x115B8, 0x115C9, 15, 0x11600, 0x11644, 15, + 0x11650, 0x11659, 15, 0x118A0, 0x118F2, 15, 0x118FF, 0x118FF, 15, 0x11AC0, 0x11AF8, 15, + 0x1236F, 0x12398, 15, 0x12463, 0x1246E, 15, 0x12474, 0x12474, 15, 0x16A40, 0x16A5E, 15, + 0x16A60, 0x16A69, 15, 0x16A6E, 0x16A6F, 15, 0x16AD0, 0x16AED, 15, 0x16AF0, 0x16AF5, 15, + 0x16B00, 0x16B45, 15, 0x16B50, 0x16B59, 15, 0x16B5B, 0x16B61, 15, 0x16B63, 0x16B77, 15, + 0x16B7D, 0x16B8F, 15, 0x1BC00, 0x1BC6A, 15, 0x1BC70, 0x1BC7C, 15, 0x1BC80, 0x1BC88, 15, + 0x1BC90, 0x1BC99, 15, 0x1BC9C, 0x1BCA3, 15, 0x1E800, 0x1E8C4, 15, 0x1E8C7, 0x1E8D6, 15, + 0x1F0BF, 0x1F0BF, 15, 0x1F0E0, 0x1F0F5, 15, 0x1F10B, 0x1F10C, 15, 0x1F321, 0x1F32C, 15, + 0x1F336, 0x1F336, 15, 0x1F37D, 0x1F37D, 15, 0x1F394, 0x1F39F, 15, 0x1F3C5, 0x1F3C5, 15, + 0x1F3CB, 0x1F3CE, 15, 0x1F3D4, 0x1F3DF, 15, 0x1F3F1, 0x1F3F7, 15, 0x1F43F, 0x1F43F, 15, + 0x1F441, 0x1F441, 15, 0x1F4F8, 0x1F4F8, 15, 0x1F4FD, 0x1F4FE, 15, 0x1F53E, 0x1F53F, 15, + 0x1F544, 0x1F54A, 15, 0x1F568, 0x1F579, 15, 0x1F57B, 0x1F5A3, 15, 0x1F5A5, 0x1F5FA, 15, + 0x1F641, 0x1F642, 15, 0x1F650, 0x1F67F, 15, 0x1F6C6, 0x1F6CF, 15, 0x1F6E0, 0x1F6EC, 15, + 0x1F6F0, 0x1F6F3, 15, 0x1F780, 0x1F7D4, 15, 0x1F800, 0x1F80B, 15, 0x1F810, 0x1F847, 15, + 0x1F850, 0x1F859, 15, 0x1F860, 0x1F887, 15, 0x1F890, 0x1F8AD, 15, 0x8B3, 0x8B4, 16, + 0x8E3, 0x8E3, 16, 0xAF9, 0xAF9, 16, 0xC5A, 0xC5A, 16, 0xD5F, 0xD5F, 16, + 0x13F5, 0x13F5, 16, 0x13F8, 0x13FD, 16, 0x20BE, 0x20BE, 16, 0x218A, 0x218B, 16, + 0x2BEC, 0x2BEF, 16, 0x9FCD, 0x9FD5, 16, 0xA69E, 0xA69E, 16, 0xA78F, 0xA78F, 16, + 0xA7B2, 0xA7B7, 16, 0xA8FC, 0xA8FD, 16, 0xAB60, 0xAB63, 16, 0xAB70, 0xABBF, 16, + 0xFE2E, 0xFE2F, 16, 0x108E0, 0x108F2, 16, 0x108F4, 0x108F5, 16, 0x108FB, 0x108FF, 16, + 0x109BC, 0x109BD, 16, 0x109C0, 0x109CF, 16, 0x109D2, 0x109FF, 16, 0x10C80, 0x10CB2, 16, + 0x10CC0, 0x10CF2, 16, 0x10CFA, 0x10CFF, 16, 0x111C9, 0x111CC, 16, 0x111DB, 0x111DF, 16, + 0x11280, 0x11286, 16, 0x11288, 0x11288, 16, 0x1128A, 0x1128D, 16, 0x1128F, 0x1129D, 16, + 0x1129F, 0x112A9, 16, 0x11300, 0x11300, 16, 0x11350, 0x11350, 16, 0x115CA, 0x115DD, 16, + 0x11700, 0x11719, 16, 0x1171D, 0x1172B, 16, 0x11730, 0x1173F, 16, 0x12399, 0x12399, 16, + 0x12480, 0x12543, 16, 0x14400, 0x14646, 16, 0x1D1DE, 0x1D1E8, 16, 0x1D800, 0x1DA8B, 16, + 0x1DA9B, 0x1DA9F, 16, 0x1DAA1, 0x1DAAF, 16, 0x1F32D, 0x1F32F, 16, 0x1F37E, 0x1F37F, 16, + 0x1F3CF, 0x1F3D3, 16, 0x1F3F8, 0x1F3FF, 16, 0x1F4FF, 0x1F4FF, 16, 0x1F54B, 0x1F54F, 16, + 0x1F643, 0x1F644, 16, 0x1F6D0, 0x1F6D0, 16, 0x1F910, 0x1F918, 16, 0x1F980, 0x1F984, 16, + 0x1F9C0, 0x1F9C0, 16, 0x2B820, 0x2CEA1, 16, 0x8B6, 0x8BD, 17, 0x8D4, 0x8E2, 17, + 0xC80, 0xC80, 17, 0xD4F, 0xD4F, 17, 0xD54, 0xD56, 17, 0xD58, 0xD5E, 17, + 0xD76, 0xD78, 17, 0x1C80, 0x1C88, 17, 0x1DFB, 0x1DFB, 17, 0x23FB, 0x23FE, 17, + 0x2E43, 0x2E44, 17, 0xA7AE, 0xA7AE, 17, 0xA8C5, 0xA8C5, 17, 0x1018D, 0x1018E, 17, + 0x104B0, 0x104D3, 17, 0x104D8, 0x104FB, 17, 0x1123E, 0x1123E, 17, 0x11400, 0x11459, 17, + 0x1145B, 0x1145B, 17, 0x1145D, 0x1145D, 17, 0x11660, 0x1166C, 17, 0x11C00, 0x11C08, 17, + 0x11C0A, 0x11C36, 17, 0x11C38, 0x11C45, 17, 0x11C50, 0x11C6C, 17, 0x11C70, 0x11C8F, 17, + 0x11C92, 0x11CA7, 17, 0x11CA9, 0x11CB6, 17, 0x16FE0, 0x16FE0, 17, 0x17000, 0x187EC, 17, + 0x18800, 0x18AF2, 17, 0x1E000, 0x1E006, 17, 0x1E008, 0x1E018, 17, 0x1E01B, 0x1E021, 17, + 0x1E023, 0x1E024, 17, 0x1E026, 0x1E02A, 17, 0x1E900, 0x1E94A, 17, 0x1E950, 0x1E959, 17, + 0x1E95E, 0x1E95F, 17, 0x1F19B, 0x1F1AC, 17, 0x1F23B, 0x1F23B, 17, 0x1F57A, 0x1F57A, 17, + 0x1F5A4, 0x1F5A4, 17, 0x1F6D1, 0x1F6D2, 17, 0x1F6F4, 0x1F6F6, 17, 0x1F919, 0x1F91E, 17, + 0x1F920, 0x1F927, 17, 0x1F930, 0x1F930, 17, 0x1F933, 0x1F93E, 17, 0x1F940, 0x1F94B, 17, + 0x1F950, 0x1F95E, 17, 0x1F985, 0x1F991, 17, 0x860, 0x86A, 18, 0x9FC, 0x9FD, 18, + 0xAFA, 0xAFF, 18, 0xD00, 0xD00, 18, 0xD3B, 0xD3C, 18, 0x1CF7, 0x1CF7, 18, + 0x1DF6, 0x1DF9, 18, 0x20BF, 0x20BF, 18, 0x23FF, 0x23FF, 18, 0x2BD2, 0x2BD2, 18, + 0x2E45, 0x2E49, 18, 0x312E, 0x312E, 18, 0x9FD6, 0x9FEA, 18, 0x1032D, 0x1032F, 18, + 0x11A00, 0x11A47, 18, 0x11A50, 0x11A83, 18, 0x11A86, 0x11A9C, 18, 0x11A9E, 0x11AA2, 18, + 0x11D00, 0x11D06, 18, 0x11D08, 0x11D09, 18, 0x11D0B, 0x11D36, 18, 0x11D3A, 0x11D3A, 18, + 0x11D3C, 0x11D3D, 18, 0x11D3F, 0x11D47, 18, 0x11D50, 0x11D59, 18, 0x16FE1, 0x16FE1, 18, + 0x1B002, 0x1B11E, 18, 0x1B170, 0x1B2FB, 18, 0x1F260, 0x1F265, 18, 0x1F6D3, 0x1F6D4, 18, + 0x1F6F7, 0x1F6F8, 18, 0x1F900, 0x1F90B, 18, 0x1F91F, 0x1F91F, 18, 0x1F928, 0x1F92F, 18, + 0x1F931, 0x1F932, 18, 0x1F94C, 0x1F94C, 18, 0x1F95F, 0x1F96B, 18, 0x1F992, 0x1F997, 18, + 0x1F9D0, 0x1F9E6, 18, 0x2CEB0, 0x2EBE0, 18, 0x560, 0x560, 19, 0x588, 0x588, 19, + 0x5EF, 0x5EF, 19, 0x7FD, 0x7FF, 19, 0x8D3, 0x8D3, 19, 0x9FE, 0x9FE, 19, + 0xA76, 0xA76, 19, 0xC04, 0xC04, 19, 0xC84, 0xC84, 19, 0x1878, 0x1878, 19, + 0x1C90, 0x1CBA, 19, 0x1CBD, 0x1CBF, 19, 0x2BBA, 0x2BBC, 19, 0x2BD3, 0x2BEB, 19, + 0x2BF0, 0x2BFE, 19, 0x2E4A, 0x2E4E, 19, 0x312F, 0x312F, 19, 0x9FEB, 0x9FEF, 19, + 0xA7AF, 0xA7AF, 19, 0xA7B8, 0xA7B9, 19, 0xA8FE, 0xA8FF, 19, 0x10A34, 0x10A35, 19, + 0x10A48, 0x10A48, 19, 0x10D00, 0x10D27, 19, 0x10D30, 0x10D39, 19, 0x10F00, 0x10F27, 19, + 0x10F30, 0x10F59, 19, 0x110CD, 0x110CD, 19, 0x11144, 0x11146, 19, 0x1133B, 0x1133B, 19, + 0x1145E, 0x1145E, 19, 0x1171A, 0x1171A, 19, 0x11800, 0x1183B, 19, 0x11A9D, 0x11A9D, 19, + 0x11D60, 0x11D65, 19, 0x11D67, 0x11D68, 19, 0x11D6A, 0x11D8E, 19, 0x11D90, 0x11D91, 19, + 0x11D93, 0x11D98, 19, 0x11DA0, 0x11DA9, 19, 0x11EE0, 0x11EF8, 19, 0x16E40, 0x16E9A, 19, + 0x187ED, 0x187F1, 19, 0x1D2E0, 0x1D2F3, 19, 0x1D372, 0x1D378, 19, 0x1EC71, 0x1ECB4, 19, + 0x1F12F, 0x1F12F, 19, 0x1F6F9, 0x1F6F9, 19, 0x1F7D5, 0x1F7D8, 19, 0x1F94D, 0x1F94F, 19, + 0x1F96C, 0x1F970, 19, 0x1F973, 0x1F976, 19, 0x1F97A, 0x1F97A, 19, 0x1F97C, 0x1F97F, 19, + 0x1F998, 0x1F9A2, 19, 0x1F9B0, 0x1F9B9, 19, 0x1F9C1, 0x1F9C2, 19, 0x1F9E7, 0x1F9FF, 19, + 0x1FA60, 0x1FA6D, 19, 0xC77, 0xC77, 20, 0xE86, 0xE86, 20, 0xE89, 0xE89, 20, + 0xE8C, 0xE8C, 20, 0xE8E, 0xE93, 20, 0xE98, 0xE98, 20, 0xEA0, 0xEA0, 20, + 0xEA8, 0xEA9, 20, 0xEAC, 0xEAC, 20, 0xEBA, 0xEBA, 20, 0x1CFA, 0x1CFA, 20, + 0x2BC9, 0x2BC9, 20, 0x2BFF, 0x2BFF, 20, 0x2E4F, 0x2E4F, 20, 0xA7BA, 0xA7BF, 20, + 0xA7C2, 0xA7C6, 20, 0xAB66, 0xAB67, 20, 0x10FE0, 0x10FF6, 20, 0x1145F, 0x1145F, 20, + 0x116B8, 0x116B8, 20, 0x119A0, 0x119A7, 20, 0x119AA, 0x119D7, 20, 0x119DA, 0x119E4, 20, + 0x11A84, 0x11A85, 20, 0x11FC0, 0x11FF1, 20, 0x11FFF, 0x11FFF, 20, 0x13430, 0x13438, 20, + 0x16F45, 0x16F4A, 20, 0x16F4F, 0x16F4F, 20, 0x16F7F, 0x16F87, 20, 0x16FE2, 0x16FE3, 20, + 0x187F2, 0x187F7, 20, 0x1B150, 0x1B152, 20, 0x1B164, 0x1B167, 20, 0x1E100, 0x1E12C, 20, + 0x1E130, 0x1E13D, 20, 0x1E140, 0x1E149, 20, 0x1E14E, 0x1E14F, 20, 0x1E2C0, 0x1E2F9, 20, + 0x1E2FF, 0x1E2FF, 20, 0x1E94B, 0x1E94B, 20, 0x1ED01, 0x1ED3D, 20, 0x1F16C, 0x1F16C, 20, + 0x1F6D5, 0x1F6D5, 20, 0x1F6FA, 0x1F6FA, 20, 0x1F7E0, 0x1F7EB, 20, 0x1F90D, 0x1F90F, 20, + 0x1F93F, 0x1F93F, 20, 0x1F971, 0x1F971, 20, 0x1F97B, 0x1F97B, 20, 0x1F9A5, 0x1F9AA, 20, + 0x1F9AE, 0x1F9AF, 20, 0x1F9BA, 0x1F9BF, 20, 0x1F9C3, 0x1F9CA, 20, 0x1F9CD, 0x1F9CF, 20, + 0x1FA00, 0x1FA53, 20, 0x1FA70, 0x1FA73, 20, 0x1FA78, 0x1FA7A, 20, 0x1FA80, 0x1FA82, 20, + 0x1FA90, 0x1FA95, 20, 0x32FF, 0x32FF, 21, 0x8BE, 0x8C7, 22, 0xB55, 0xB55, 22, + 0xD04, 0xD04, 22, 0xD81, 0xD81, 22, 0x1ABF, 0x1AC0, 22, 0x2B97, 0x2B97, 22, + 0x2E50, 0x2E52, 22, 0x31BB, 0x31BF, 22, 0x4DB6, 0x4DBF, 22, 0x9FF0, 0x9FFC, 22, + 0xA7C7, 0xA7CA, 22, 0xA7F5, 0xA7F6, 22, 0xA82C, 0xA82C, 22, 0xAB68, 0xAB6B, 22, + 0x1019C, 0x1019C, 22, 0x10E80, 0x10EA9, 22, 0x10EAB, 0x10EAD, 22, 0x10EB0, 0x10EB1, 22, + 0x10FB0, 0x10FCB, 22, 0x11147, 0x11147, 22, 0x111CE, 0x111CF, 22, 0x1145A, 0x1145A, 22, + 0x11460, 0x11461, 22, 0x11900, 0x11906, 22, 0x11909, 0x11909, 22, 0x1190C, 0x11913, 22, + 0x11915, 0x11916, 22, 0x11918, 0x11935, 22, 0x11937, 0x11938, 22, 0x1193B, 0x11946, 22, + 0x11950, 0x11959, 22, 0x11FB0, 0x11FB0, 22, 0x16FE4, 0x16FE4, 22, 0x16FF0, 0x16FF1, 22, + 0x18AF3, 0x18CD5, 22, 0x18D00, 0x18D08, 22, 0x1F10D, 0x1F10F, 22, 0x1F16D, 0x1F16F, 22, + 0x1F1AD, 0x1F1AD, 22, 0x1F6D6, 0x1F6D7, 22, 0x1F6FB, 0x1F6FC, 22, 0x1F8B0, 0x1F8B1, 22, + 0x1F90C, 0x1F90C, 22, 0x1F972, 0x1F972, 22, 0x1F977, 0x1F978, 22, 0x1F9A3, 0x1F9A4, 22, + 0x1F9AB, 0x1F9AD, 22, 0x1F9CB, 0x1F9CB, 22, 0x1FA74, 0x1FA74, 22, 0x1FA83, 0x1FA86, 22, + 0x1FA96, 0x1FAA8, 22, 0x1FAB0, 0x1FAB6, 22, 0x1FAC0, 0x1FAC2, 22, 0x1FAD0, 0x1FAD6, 22, + 0x1FB00, 0x1FB92, 22, 0x1FB94, 0x1FBCA, 22, 0x1FBF0, 0x1FBF9, 22, 0x2A6D7, 0x2A6DD, 22, + 0x30000, 0x3134A, 22, 0x61D, 0x61D, 23, 0x870, 0x88E, 23, 0x890, 0x891, 23, + 0x898, 0x89F, 23, 0x8B5, 0x8B5, 23, 0x8C8, 0x8D2, 23, 0xC3C, 0xC3C, 23, + 0xC5D, 0xC5D, 23, 0xCDD, 0xCDD, 23, 0x170D, 0x170D, 23, 0x1715, 0x1715, 23, + 0x171F, 0x171F, 23, 0x180F, 0x180F, 23, 0x1AC1, 0x1ACE, 23, 0x1B4C, 0x1B4C, 23, + 0x1B7D, 0x1B7E, 23, 0x1DFA, 0x1DFA, 23, 0x20C0, 0x20C0, 23, 0x2C2F, 0x2C2F, 23, + 0x2C5F, 0x2C5F, 23, 0x2E53, 0x2E5D, 23, 0x9FFD, 0x9FFF, 23, 0xA7C0, 0xA7C1, 23, + 0xA7D0, 0xA7D1, 23, 0xA7D3, 0xA7D3, 23, 0xA7D5, 0xA7D9, 23, 0xA7F2, 0xA7F4, 23, + 0xFBC2, 0xFBC2, 23, 0xFD40, 0xFD4F, 23, 0xFDCF, 0xFDCF, 23, 0xFDFE, 0xFDFF, 23, + 0x10570, 0x1057A, 23, 0x1057C, 0x1058A, 23, 0x1058C, 0x10592, 23, 0x10594, 0x10595, 23, + 0x10597, 0x105A1, 23, 0x105A3, 0x105B1, 23, 0x105B3, 0x105B9, 23, 0x105BB, 0x105BC, 23, + 0x10780, 0x10785, 23, 0x10787, 0x107B0, 23, 0x107B2, 0x107BA, 23, 0x10F70, 0x10F89, 23, + 0x11070, 0x11075, 23, 0x110C2, 0x110C2, 23, 0x116B9, 0x116B9, 23, 0x11740, 0x11746, 23, + 0x11AB0, 0x11ABF, 23, 0x12F90, 0x12FF2, 23, 0x16A70, 0x16ABE, 23, 0x16AC0, 0x16AC9, 23, + 0x1AFF0, 0x1AFF3, 23, 0x1AFF5, 0x1AFFB, 23, 0x1AFFD, 0x1AFFE, 23, 0x1B11F, 0x1B122, 23, + 0x1CF00, 0x1CF2D, 23, 0x1CF30, 0x1CF46, 23, 0x1CF50, 0x1CFC3, 23, 0x1D1E9, 0x1D1EA, 23, + 0x1DF00, 0x1DF1E, 23, 0x1E290, 0x1E2AE, 23, 0x1E7E0, 0x1E7E6, 23, 0x1E7E8, 0x1E7EB, 23, + 0x1E7ED, 0x1E7EE, 23, 0x1E7F0, 0x1E7FE, 23, 0x1F6DD, 0x1F6DF, 23, 0x1F7F0, 0x1F7F0, 23, + 0x1F979, 0x1F979, 23, 0x1F9CC, 0x1F9CC, 23, 0x1FA7B, 0x1FA7C, 23, 0x1FAA9, 0x1FAAC, 23, + 0x1FAB7, 0x1FABA, 23, 0x1FAC3, 0x1FAC5, 23, 0x1FAD7, 0x1FAD9, 23, 0x1FAE0, 0x1FAE7, 23, + 0x1FAF0, 0x1FAF6, 23, 0x2A6DE, 0x2A6DF, 23, 0x2B735, 0x2B738, 23, 0xCF3, 0xCF3, 24, + 0xECE, 0xECE, 24, 0x10EFD, 0x10EFF, 24, 0x1123F, 0x11241, 24, 0x11B00, 0x11B09, 24, + 0x11F00, 0x11F10, 24, 0x11F12, 0x11F3A, 24, 0x11F3E, 0x11F59, 24, 0x1342F, 0x1342F, 24, + 0x13439, 0x13455, 24, 0x1B132, 0x1B132, 24, 0x1B155, 0x1B155, 24, 0x1D2C0, 0x1D2D3, 24, + 0x1DF25, 0x1DF2A, 24, 0x1E030, 0x1E06D, 24, 0x1E08F, 0x1E08F, 24, 0x1E4D0, 0x1E4F9, 24, + 0x1F6DC, 0x1F6DC, 24, 0x1F774, 0x1F776, 24, 0x1F77B, 0x1F77F, 24, 0x1F7D9, 0x1F7D9, 24, + 0x1FA75, 0x1FA77, 24, 0x1FA87, 0x1FA88, 24, 0x1FAAD, 0x1FAAF, 24, 0x1FABB, 0x1FABD, 24, + 0x1FABF, 0x1FABF, 24, 0x1FACE, 0x1FACF, 24, 0x1FADA, 0x1FADB, 24, 0x1FAE8, 0x1FAE8, 24, + 0x1FAF7, 0x1FAF8, 24, 0x2B739, 0x2B739, 24, 0x31350, 0x323AF, 24, 0x2FFC, 0x2FFF, 25, + 0x31EF, 0x31EF, 25, 0x2EBF0, 0x2EE5D, 25, 0x897, 0x897, 26, 0x1B4E, 0x1B4F, 26, + 0x1B7F, 0x1B7F, 26, 0x1C89, 0x1C8A, 26, 0x2427, 0x2429, 26, 0x31E4, 0x31E5, 26, + 0xA7CB, 0xA7CD, 26, 0xA7DA, 0xA7DC, 26, 0x105C0, 0x105F3, 26, 0x10D40, 0x10D65, 26, + 0x10D69, 0x10D85, 26, 0x10D8E, 0x10D8F, 26, 0x10EC2, 0x10EC4, 26, 0x10EFC, 0x10EFC, 26, + 0x11380, 0x11389, 26, 0x1138B, 0x1138B, 26, 0x1138E, 0x1138E, 26, 0x11390, 0x113B5, 26, + 0x113B7, 0x113C0, 26, 0x113C2, 0x113C2, 26, 0x113C5, 0x113C5, 26, 0x113C7, 0x113CA, 26, + 0x113CC, 0x113D5, 26, 0x113D7, 0x113D8, 26, 0x113E1, 0x113E2, 26, 0x116D0, 0x116E3, 26, + 0x11BC0, 0x11BE1, 26, 0x11BF0, 0x11BF9, 26, 0x11F5A, 0x11F5A, 26, 0x13460, 0x143FA, 26, + 0x16100, 0x16139, 26, 0x16D40, 0x16D79, 26, 0x18CFF, 0x18CFF, 26, 0x1CC00, 0x1CCF9, 26, + 0x1CD00, 0x1CEB3, 26, 0x1E5D0, 0x1E5FA, 26, 0x1E5FF, 0x1E5FF, 26, 0x1F8B2, 0x1F8BB, 26, + 0x1F8C0, 0x1F8C1, 26, 0x1FA89, 0x1FA89, 26, 0x1FA8F, 0x1FA8F, 26, 0x1FABE, 0x1FABE, 26, + 0x1FAC6, 0x1FAC6, 26, 0x1FADC, 0x1FADC, 26, 0x1FADF, 0x1FADF, 26, 0x1FAE9, 0x1FAE9, 26, + 0x1FBCB, 0x1FBEF, 26, 0x88F, 0x88F, 27, 0xC5C, 0xC5C, 27, 0xCDC, 0xCDC, 27, + 0x1ACF, 0x1ADD, 27, 0x1AE0, 0x1AEB, 27, 0x20C1, 0x20C1, 27, 0x2B96, 0x2B96, 27, + 0xA7CE, 0xA7CF, 27, 0xA7D2, 0xA7D2, 27, 0xA7D4, 0xA7D4, 27, 0xA7F1, 0xA7F1, 27, + 0xFBC3, 0xFBD2, 27, 0xFD90, 0xFD91, 27, 0xFDC8, 0xFDCE, 27, 0x10940, 0x10959, 27, + 0x10EC5, 0x10EC7, 27, 0x10ED0, 0x10ED8, 27, 0x10EFA, 0x10EFB, 27, 0x11B60, 0x11B67, 27, + 0x11DB0, 0x11DDB, 27, 0x11DE0, 0x11DE9, 27, 0x16EA0, 0x16EB8, 27, 0x16EBB, 0x16ED3, 27, + 0x16FF2, 0x16FF6, 27, 0x187F8, 0x187FF, 27, 0x18D09, 0x18D1E, 27, 0x18D80, 0x18DF2, 27, + 0x1CCFA, 0x1CCFC, 27, 0x1CEBA, 0x1CED0, 27, 0x1CEE0, 0x1CEF0, 27, 0x1E6C0, 0x1E6DE, 27, + 0x1E6E0, 0x1E6F5, 27, 0x1E6FE, 0x1E6FF, 27, 0x1F6D8, 0x1F6D8, 27, 0x1F777, 0x1F77A, 27, + 0x1F8D0, 0x1F8D8, 27, 0x1FA54, 0x1FA57, 27, 0x1FA8A, 0x1FA8A, 27, 0x1FA8E, 0x1FA8E, 27, + 0x1FAC8, 0x1FAC8, 27, 0x1FACD, 0x1FACD, 27, 0x1FAEA, 0x1FAEA, 27, 0x1FAEF, 0x1FAEF, 27, + 0x1FBFA, 0x1FBFA, 27, 0x2B73A, 0x2B73F, 27, 0x2CEA2, 0x2CEAD, 27, 0x323B0, 0x33479, 27, + }; + + private static final UnicodeSet[] EXACT_SETS = buildExactSets(); + private static final UnicodeSet[] CUMULATIVE_SETS = buildCumulativeSets(); + private static final UnicodeSet UNASSIGNED = new UnicodeSet(0, 0x10ffff) + .removeAll(CUMULATIVE_SETS[CUMULATIVE_SETS.length - 1]).freeze(); + + static UnicodeSet exactSet(String version) { + int versionIndex = versionIndex(version); + return versionIndex < 0 ? null : EXACT_SETS[versionIndex]; + } + + static UnicodeSet cumulativeSet(String version) { + int versionIndex = versionIndex(version); + return versionIndex < 0 ? null : CUMULATIVE_SETS[versionIndex]; + } + + static UnicodeSet unassignedSet() { + return UNASSIGNED; + } + + private static UnicodeSet[] buildExactSets() { + UnicodeSet[] sets = new UnicodeSet[VERSIONS.length]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + for (int i = 0; i < RANGES.length; i += 3) { + sets[RANGES[i + 2]].add(RANGES[i], RANGES[i + 1]); + } + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + + private static UnicodeSet[] buildCumulativeSets() { + UnicodeSet[] sets = new UnicodeSet[VERSIONS.length]; + UnicodeSet running = new UnicodeSet(); + for (int i = 0; i < sets.length; i++) { + running.addAll(EXACT_SETS[i]); + sets[i] = new UnicodeSet(running).freeze(); + } + return sets; + } + + private static int versionIndex(String version) { + for (int i = 0; i < VERSIONS.length; i++) { + if (VERSIONS[i].equals(version)) return i; + } + return -1; + } + + private PerlUnicodeAgeData() { + } +} diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java index 66491c48eb..bec8aab452 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java @@ -178,10 +178,9 @@ static void updateControlVerbVariables(String mark, String error) { Pattern patternUnicodeNoInternalMarkers; // "Notempty" variant patterns for zero-length match guard retry. // In Perl, after a zero-length /gc match at position P, the next attempt - // stays at P but uses NOTEMPTY (forbidding zero-length results, causing - // backtracking from lazy quantifiers like ??). Java lacks this, so we - // compile a variant where ?? is converted to ? (greedy) and (?=[\s\S]) - // is prepended to prevent matching at end of string. + // stays at P but uses NOTEMPTY. Java lacks this option, so an opaque region + // starting at P plus a trailing one-character lookbehind requires the + // selected alternative to consume input. Pattern notemptyPattern; Pattern notemptyPatternUnicode; JoniRegexPattern recursivePattern; @@ -722,20 +721,19 @@ private static synchronized RuntimeRegex compileSynchronized( } } - // Compile "notempty" variant for /g patterns. - // This is used after a zero-length match to retry at the same position - // with a regex that prefers non-zero-length matches (like Perl's NOTEMPTY). - // Transform: prepend (?=[\s\S]) and convert ?? to ? (lazy→greedy). + // Compile a "notempty" variant for /g patterns. With an opaque + // region starting at the retry position, the trailing lookbehind + // succeeds only when the selected alternative consumed input. if (!usesRecursiveBackend && regex.regexFlags.isGlobalMatch() && javaPattern != null) { try { - String notemptyJava = "(?=[\\s\\S])" + javaPattern.replace("??", "?"); + String notemptyJava = "(?:" + javaPattern + ")(?<=[\\s\\S])"; regex.notemptyPattern = Pattern.compile(notemptyJava, regex.patternFlags); if (regex.patternFlagsUnicode != regex.patternFlags) { String javaPatternUnicode = preProcessRegex(compilePatternString, regex.regexFlags.with("u", "a"), false); - String notemptyUnicode = "(?=[\\s\\S])" + javaPatternUnicode + String notemptyUnicode = javaPatternUnicode .replace("\\p{Punct}", "[\\p{P}\\p{S}]") - .replace("\\P{Punct}", "[^\\p{P}\\p{S}]") - .replace("??", "?"); + .replace("\\P{Punct}", "[^\\p{P}\\p{S}]"); + notemptyUnicode = "(?:" + notemptyUnicode + ")(?<=[\\s\\S])"; regex.notemptyPatternUnicode = Pattern.compile(notemptyUnicode, regex.patternFlagsUnicode); } else { regex.notemptyPatternUnicode = regex.notemptyPattern; @@ -2022,33 +2020,13 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc if (RuntimePosLvalue.hadZeroLengthMatchAt( string, posScalar.getInt(), patternKey)) { // First, try the notempty variant at the SAME position (Perl behavior) - boolean notemptySucceeded = false; - if (regex.notemptyPattern != null) { - // Select the right notempty pattern variant (byte/unicode) - Pattern notemptyPat = regex.notemptyPattern; - if (regex.notemptyPatternUnicode != null && regex.notemptyPatternUnicode != regex.notemptyPattern) { - if (!(regex.regexFlags != null && regex.regexFlags.isAscii()) - && !hasInlineAsciiModifier(regex.patternString) - && Utf8.isUtf8(string)) { - notemptyPat = regex.notemptyPatternUnicode; - } - } - RegexMatcher notemptyMatcher = new JavaRegexMatcher( - notemptyPat.matcher(matchInput), regex.branchResetCaptureMap); - notemptyMatcher.region(startPos, inputStr.length()); - if (notemptyMatcher.find()) { - // Check \G constraint: match must start at startPos - if (!regex.useGAssertion || notemptyMatcher.start() == startPos) { - // Verify it's actually non-zero-length - if (notemptyMatcher.end() > notemptyMatcher.start()) { - // Success! Use the notempty matcher's result - matcher = notemptyMatcher; - skipFirstFind = true; - notemptySucceeded = true; - RuntimePosLvalue.recordNonZeroLengthMatch(string); - } - } - } + RegexMatcher notemptyMatcher = findNonEmptyGlobalRetry( + regex, string, inputStr, matchInput, startPos); + boolean notemptySucceeded = notemptyMatcher != null; + if (notemptySucceeded) { + matcher = notemptyMatcher; + skipFirstFind = true; + RuntimePosLvalue.recordNonZeroLengthMatch(string); } if (!notemptySucceeded) { @@ -2112,7 +2090,7 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc int capture = 1; int previousPos = startPos; // Track the previous position - int previousMatchEnd = -1; // Track end of previous match + int previousZeroLengthStart = -1; boolean previousMatchUsedPFlag = regexState.lastMatchUsedPFlag; // Inline /p is Perl match-variable policy rather than a Joni option. // Publish it while callouts execute; restore the preceding successful @@ -2183,10 +2161,11 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc // Update the position for the next match int matchStart = matcher.start(); int matchEnd = matcher.end(); + boolean zeroLengthMatch = matchEnd == matchStart; boolean forcedAdvance = false; // Detect zero-length match that would cause infinite loop - if (matchEnd == matchStart && matchStart == previousMatchEnd) { + if (zeroLengthMatch && matchStart == previousZeroLengthStart) { // Consecutive zero-length match at same position - advance by 1 or stop if (matchEnd >= inputStr.length()) { // At end of string, stop matching @@ -2197,7 +2176,7 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc forcedAdvance = true; } - previousMatchEnd = matchEnd; + previousZeroLengthStart = zeroLengthMatch ? matchStart : -1; if (ctx == RuntimeContextType.SCALAR || ctx == RuntimeContextType.VOID) { // Set pos to the end of the current match to prepare for the next search @@ -2233,6 +2212,16 @@ private static RuntimeBase matchRegexDirect(RuntimeScalar quotedRegex, RuntimeSc matcher.region(startPos, inputStr.length()); matcher.useAnchoringBounds(false); } + if (zeroLengthMatch) { + RegexMatcher notemptyMatcher = findNonEmptyGlobalRetry( + regex, string, inputStr, matchInput, startPos); + if (notemptyMatcher != null) { + matcher = notemptyMatcher; + skipFirstFind = true; + nativeGlobalPosition = regex.useGAssertion + && matcher.setGlobalPosition(startPos); + } + } } } @@ -2697,6 +2686,55 @@ private static int bumpGlobalMatchPosition(String inputStr, int offset) { return offset + Character.charCount(inputStr.codePointAt(offset)); } + private static RegexMatcher findNonEmptyGlobalRetry(RuntimeRegex regex, + RuntimeScalar string, + String inputStr, + CharSequence matchInput, + int startPos) { + RegexMatcher retryMatcher; + boolean nativeNotEmpty; + if (regex.recursivePattern != null) { + retryMatcher = regex.selectRecursivePattern(string) + .matcher(inputStr, regex.executableCallbacks, string); + nativeNotEmpty = true; + } else { + Pattern notemptyPattern = regex.notemptyPattern; + if (regex.notemptyPatternUnicode != null + && regex.notemptyPatternUnicode != regex.notemptyPattern + && !(regex.regexFlags != null && regex.regexFlags.isAscii()) + && !hasInlineAsciiModifier(regex.patternString) + && Utf8.isUtf8(string)) { + notemptyPattern = regex.notemptyPatternUnicode; + } + if (notemptyPattern == null) { + // A compiled qr// can be created without /g and interpolated + // later into a global match. Build its retry form lazily from + // the selected byte/Unicode Java pattern. + notemptyPattern = compileNonEmptySubstitutionPattern( + regex.selectPattern(string, inputStr)); + if (notemptyPattern == null) { + return null; + } + } + retryMatcher = new JavaRegexMatcher( + notemptyPattern.matcher(matchInput), regex.branchResetCaptureMap); + nativeNotEmpty = false; + } + + retryMatcher.region(startPos, inputStr.length()); + retryMatcher.useAnchoringBounds(false); + if (regex.useGAssertion) { + retryMatcher.setGlobalPosition(startPos); + } + boolean found = nativeNotEmpty + ? retryMatcher.findNotEmpty() + : retryMatcher.find(); + return found + && retryMatcher.start() == startPos + && retryMatcher.end() > startPos + ? retryMatcher : null; + } + private static void setSubstitutionRegion(RegexMatcher matcher, int start, int end, boolean transparentBounds) { matcher.region(start, end); // The substitution loop drives the matcher by changing regions. Keep @@ -3209,11 +3247,10 @@ public static RuntimeScalar makeMatchResultScalar(String value) { public static RuntimeScalar matcherStart(int group) { if (group == 0) { - return state().lastMatchStart >= 0 ? getScalarInt(state().lastMatchStart) : scalarUndef; + return publicMatcherOffset(state().lastMatchStart); } if (state().manualCaptureStarts != null && group > 0 && group <= state().manualCaptureStarts.length) { - int start = state().manualCaptureStarts[group - 1]; - return start >= 0 ? getScalarInt(start) : scalarUndef; + return publicMatcherOffset(state().manualCaptureStarts[group - 1]); } if (state().globalMatcher == null) { return scalarUndef; @@ -3228,7 +3265,7 @@ public static RuntimeScalar matcherStart(int group) { if (start == -1) { return scalarUndef; } - return getScalarInt(start); + return publicMatcherOffset(start); } catch (IllegalStateException e) { return scalarUndef; } @@ -3236,11 +3273,10 @@ public static RuntimeScalar matcherStart(int group) { public static RuntimeScalar matcherEnd(int group) { if (group == 0) { - return state().lastMatchEnd >= 0 ? getScalarInt(state().lastMatchEnd) : scalarUndef; + return publicMatcherOffset(state().lastMatchEnd); } if (state().manualCaptureEnds != null && group > 0 && group <= state().manualCaptureEnds.length) { - int end = state().manualCaptureEnds[group - 1]; - return end >= 0 ? getScalarInt(end) : scalarUndef; + return publicMatcherOffset(state().manualCaptureEnds[group - 1]); } if (state().globalMatcher == null) { return scalarUndef; @@ -3255,12 +3291,24 @@ public static RuntimeScalar matcherEnd(int group) { if (end == -1) { return scalarUndef; } - return getScalarInt(end); + return publicMatcherOffset(end); } catch (IllegalStateException e) { return scalarUndef; } } + private static RuntimeScalar publicMatcherOffset(int matcherOffset) { + RuntimeRegexState regexState = state(); + if (matcherOffset < 0) { + return scalarUndef; + } + if (regexState.lastMatchWasByteString || regexState.globalMatchString == null) { + return getScalarInt(matcherOffset); + } + return getScalarInt(PerlUtfString.perlOffsetForJavaIndex( + regexState.globalMatchString, matcherOffset)); + } + public static int matcherSize() { if (state().manualCaptureStarts != null) { return state().manualCaptureStarts.length + 1; diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index 8957f8feb7..727f1556cb 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -18,12 +18,6 @@ public class UnicodeResolver { private static final Pattern USER_DEFINED_PROPERTY_NAME = Pattern.compile( "^(?:[A-Za-z_][A-Za-z0-9_]*::)*(?:Is|In)[A-Za-z_][A-Za-z0-9_]*$"); - private static final String[] PERL_UNICODE_AGE_VERSIONS = { - "1.1", "2.0", "2.1", "3.0", "3.1", "3.2", "4.0", "4.1", - "5.0", "5.1", "5.2", "6.0", "6.1", "6.2", "6.3", "7.0", - "8.0", "9.0", "10.0", "11.0", "12.0", "12.1", "13.0", - "14.0", "15.0" - }; /** * Cache for user-defined property subroutine results. * Perl only calls user-defined property subs once per unique name and caches the result. @@ -1050,10 +1044,12 @@ private static int unicodePropertyValue(int property, String alias) { } private static String translatePerlAgeProperty(String property, boolean negated) { - int equals = property.indexOf('='); - if (equals <= 0 || equals == property.length() - 1) return null; + int delimiter = property.indexOf('='); + int colon = property.indexOf(':'); + if (delimiter < 0 || colon > 0 && colon < delimiter) delimiter = colon; + if (delimiter <= 0 || delimiter == property.length() - 1) return null; - String name = property.substring(0, equals) + String name = property.substring(0, delimiter) .replace("_", "").replace("-", "").replace(" ", ""); boolean exact; if (name.equalsIgnoreCase("Age")) { @@ -1064,49 +1060,48 @@ private static String translatePerlAgeProperty(String property, boolean negated) return null; } - String requested = normalizeUnicodeAgeVersion(property.substring(equals + 1)); + String requested = normalizeUnicodeAgeVersion(property.substring(delimiter + 1)); if (requested.equalsIgnoreCase("NA") || requested.equalsIgnoreCase("Unassigned")) { - UnicodeSet unassigned = new UnicodeSet(); - unassigned.applyPropertyAlias("Age", "Unassigned"); - return wrapCharClass(unicodeSetToJavaPattern(unassigned), negated); + return wrapCharClass( + unicodeSetToJavaPattern(PerlUnicodeAgeData.unassignedSet()), negated); } - int versionIndex = -1; - for (int i = 0; i < PERL_UNICODE_AGE_VERSIONS.length; i++) { - if (PERL_UNICODE_AGE_VERSIONS[i].equals(requested)) { - versionIndex = i; - break; - } - } - if (versionIndex < 0) { + UnicodeSet result = exact + ? PerlUnicodeAgeData.exactSet(requested) + : PerlUnicodeAgeData.cumulativeSet(requested); + if (result == null) { throw new IllegalArgumentException("Unsupported Unicode age version: " + requested); } - - UnicodeSet result = cumulativeAgeSet(PERL_UNICODE_AGE_VERSIONS[versionIndex]); - if (exact && versionIndex > 0) { - result.removeAll(cumulativeAgeSet(PERL_UNICODE_AGE_VERSIONS[versionIndex - 1])); - } return wrapCharClass(unicodeSetToJavaPattern(result), negated); } private static String normalizeUnicodeAgeVersion(String value) { - String normalized = value.trim().replace("_", "").replace(" ", ""); + String normalized = value.trim(); + if (normalized.startsWith(":\\A") && normalized.endsWith("\\z:") + && normalized.length() > 6) { + normalized = normalized.substring(3, normalized.length() - 3); + } + normalized = normalized.replaceAll("[\\s_+\\-]", ""); + if (normalized.equalsIgnoreCase("NA") + || normalized.equalsIgnoreCase("Unassigned")) return normalized; if (normalized.length() > 1 && (normalized.charAt(0) == 'v' || normalized.charAt(0) == 'V')) { normalized = normalized.substring(1); if (!normalized.contains(".")) { + if (normalized.length() < 2) return normalized; normalized = normalized.substring(0, normalized.length() - 1) + "." + normalized.charAt(normalized.length() - 1); } } if (!normalized.contains(".")) normalized += ".0"; - return normalized; - } - - private static UnicodeSet cumulativeAgeSet(String version) { - UnicodeSet set = new UnicodeSet(); - set.applyPropertyAlias("Age", version); - return set; + String[] parts = normalized.split("\\.", -1); + if (parts.length != 2 || !parts[0].matches("[0-9]+") + || !parts[1].matches("[0-9]+")) return normalized; + try { + return Integer.parseInt(parts[0]) + "." + Integer.parseInt(parts[1]); + } catch (NumberFormatException invalidVersion) { + return normalized; + } } // Helper method to get XID_Start pattern using ICU4J diff --git a/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeAgeDataTest.java b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeAgeDataTest.java new file mode 100644 index 0000000000..a32a8805bc --- /dev/null +++ b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeAgeDataTest.java @@ -0,0 +1,20 @@ +package org.perlonjava.runtime.regex; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; + +@Tag("unit") +class PerlUnicodeAgeDataTest { + @Test + void usesPinnedPerl544Unicode17Data() { + assertEquals("17.0.0", PerlUnicodeAgeData.UNICODE_VERSION); + assertTrue(PerlUnicodeAgeData.exactSet("17.0").contains(0x33479)); + assertFalse(PerlUnicodeAgeData.exactSet("17.0").contains(0x3347a)); + assertTrue(PerlUnicodeAgeData.cumulativeSet("17.0").contains('A')); + assertTrue(PerlUnicodeAgeData.unassignedSet().contains(0x3347a)); + } +} diff --git a/src/test/resources/unit/regex/unicode_age_loose_pinned.t b/src/test/resources/unit/regex/unicode_age_loose_pinned.t new file mode 100644 index 0000000000..b7c730ea25 --- /dev/null +++ b/src/test/resources/unit/regex/unicode_age_loose_pinned.t @@ -0,0 +1,26 @@ +use strict; +use warnings; +use utf8; +use Test::More tests => 14; +no warnings 'experimental::uniprop_wildcards'; + +my $unicode_16 = chr 0x1FBEF; + +ok($unicode_16 =~ /\p{Age=v160}/, 'compact V age alias'); +ok($unicode_16 =~ /\p{Age= +0000016.0}/, 'loose plus and leading zero age alias'); +ok($unicode_16 =~ /\p{Age=-V16_0}/, 'loose hyphen and underscore age alias'); +ok($unicode_16 =~ /\p{Is_Age=0000016.0}/, 'Is_Age accepts a loose numeric value'); +ok($unicode_16 =~ /\p{In=v160}/, 'In accepts a compact V value'); +ok($unicode_16 =~ /\p{Present_In=16.0}/, 'Present_In includes the introduction version'); +ok('A' =~ /\p{In=16.0}/, 'In is cumulative across older versions'); +ok('A' !~ /\p{Age=16.0}/, 'Age remains an exact introduction version'); +ok($unicode_16 =~ /\p{Age=:\AV16_0\z:}/, + 'anchored Unicode property wildcard resolves a long age value'); +ok($unicode_16 =~ /\p{Age=:\Av160\z:}/, + 'anchored Unicode property wildcard resolves a compact age value'); +ok($unicode_16 =~ /\p{Age: v160}/, 'colon delimits an Age property value'); +ok('A' =~ /\p{Present_In: 16.0}/, + 'colon delimits a cumulative Present_In value'); +ok(chr(0x3347A) =~ /\p{Age=Unassigned}/, + 'Unassigned uses the pinned complement'); +ok(chr(0x3347A) =~ /\p{Is_Age=NA}/, 'NA aliases the unassigned age value'); diff --git a/src/test/resources/unit/regex/zero_width_global_retry.t b/src/test/resources/unit/regex/zero_width_global_retry.t new file mode 100644 index 0000000000..a4c863ee77 --- /dev/null +++ b/src/test/resources/unit/regex/zero_width_global_retry.t @@ -0,0 +1,213 @@ +use strict; +use warnings; +use utf8; +use Test::More; + +sub value { defined $_[0] ? $_[0] : '' } + +my $rx = qr/(?:(?)|(?.))/; + +sub collect { + my ($s, $pattern) = @_; + my @events; + pos($s) = 0; + while ($s =~ /$pattern/g) { + push @events, [ + value($+{empty}), + value($+{char}), + $-[0], + $+[0], + pos($s), + ]; + } + return (\@events, pos($s)); +} + +my ($ascii, $ascii_pos) = collect('ab', $rx); +is_deeply( + $ascii, + [ + ['', '', 0, 0, 0], + ['', 'a', 0, 1, 1], + ['', '', 1, 1, 1], + ['', 'b', 1, 2, 2], + ['', '', 2, 2, 2], + ], + 'one empty then one consuming match at each ASCII position', +); +ok(!defined($ascii_pos), 'exhausted non-c global match resets pos'); + +my ($unicode, $unicode_pos) = collect("\x{1F642}a", $rx); +is_deeply( + $unicode, + [ + ['', '', 0, 0, 0], + ['', "\x{1F642}", 0, 1, 1], + ['', '', 1, 1, 1], + ['', 'a', 1, 2, 2], + ['', '', 2, 2, 2], + ], + 'Unicode spans and pos are character offsets', +); +ok(!defined($unicode_pos), 'exhausted Unicode global match resets pos'); + +my $captured_unicode = "\x{1F642}a"; +ok($captured_unicode =~ /(\x{1F642})(a)/, + 'non-global capture matches supplementary and BMP characters'); +is_deeply([@-], [0, 0, 1], + 'non-global capture starts are Perl character offsets'); +is_deeply([@+], [2, 1, 2], + 'non-global capture ends are Perl character offsets'); + +my $replacement_unicode = "\x{1F642}a"; +my @replacement_offsets; +$replacement_unicode =~ s{(.)}{do { + push @replacement_offsets, [$-[0], $+[0]]; + $1; +}}eg; +is_deeply( + \@replacement_offsets, + [[0, 1], [1, 2]], + 'Unicode substitution callbacks receive Perl character offsets', +); + +my $byte_subject = "\xF0\x9F\x99\x82"; +my @byte_offsets; +{ + use bytes; + $byte_subject =~ /(....)/; + @byte_offsets = ($-[0], $+[0], $-[1], $+[1]); +} +is_deeply( + \@byte_offsets, + [0, 4, 0, 4], + 'byte matches retain byte offsets', +); + +my $list_rx = qr/(?:|.)/; +my @list_matches = ('ab' =~ /$list_rx/g); +is_deeply( + \@list_matches, + ['', 'a', '', 'b', ''], + 'list-context global match retries consuming alternatives at the same position', +); + +my $list_capture_rx = qr/((?:)|.)/; +my @list_captures = ('ab' =~ /$list_capture_rx/g); +is_deeply( + \@list_captures, + ['', 'a', '', 'b', ''], + 'list-context global captures publish every empty and consuming result', +); + +my @artificial_anchor = ('ab' =~ /(?:|^b)/g); +is_deeply( + \@artificial_anchor, + ['', '', ''], + 'global retry does not treat its artificial region start as string start', +); + +my @multiline_anchor = ("a\nb" =~ /(?:|^b)/mg); +is_deeply( + \@multiline_anchor, + ['', '', '', 'b', ''], + 'global retry preserves a real multiline start anchor', +); + +my $multi_rx = qr/(?:(?)|(?(?=a))|(?a))/; +my $multi = 'a'; +my @branches; +while ($multi =~ /$multi_rx/g) { + my $branch = defined($+{first}) ? 'first' + : defined($+{second}) ? 'second' + : 'letter'; + push @branches, [$branch, $-[0], $+[0], pos($multi)]; +} +is_deeply( + \@branches, + [ + ['first', 0, 0, 0], + ['letter', 0, 1, 1], + ['first', 1, 1, 1], + ], + 'same-position retry suppresses every second empty alternative', +); + +my $state = 'a'; +pos($state) = 0; +my @gc; +for (1 .. 4) { + my $ok = $state =~ /$rx/gc; + push @gc, $ok + ? [ + 1, + value($+{empty}), + value($+{char}), + $-[0], + $+[0], + pos($state), + ] + : [0, pos($state)]; +} +is_deeply( + \@gc, + [ + [1, '', '', 0, 0, 0], + [1, '', 'a', 0, 1, 1], + [1, '', '', 1, 1, 1], + [0, 1], + ], + 'gc preserves pos after failure following terminal empty match', +); +ok(!($state =~ /$rx/g), 'non-c retry at preserved terminal pos fails'); +ok(!defined(pos($state)), 'non-c failure resets pos'); +ok( + $state =~ /$rx/g + && defined($+{empty}) + && $+{empty} eq '' + && pos($state) == 0, + 'after reset global matching restarts with empty match at zero', +); + +my $sub_rx = qr/((?:)|.)/; +my $sub = 'a'; +my @sub_events; +my $sub_count = $sub =~ s{$sub_rx}{do { + push @sub_events, [$1, $-[0], $+[0]]; + '<' . $1 . '>'; +}}eg; +is($sub_count, 3, 'substitution performs empty consuming empty sequence'); +is($sub, '<><>', 'substitution preserves all three replacements in order'); +is_deeply( + \@sub_events, + [ + ['', 0, 0], + ['a', 0, 1], + ['', 1, 1], + ], + 'substitution callbacks observe original same-position spans', +); + +my $quoted = qr/(?:'[^']*'|"[^"]*")*/; +my $sql = q{SELECT '(??)', (??)}; +my $replacements = 0; +$sql =~ s[($quoted|\(\?\?\))] { + $1 eq '(??)' + ? do { + ++$replacements; + '(?, ?)'; + } + : $1 +}eg; +is( + $sql, + q{SELECT '(??)', (?, ?)}, + 'DBIx substitution leaves quoted marker and replaces unquoted marker', +); +is( + $replacements, + 1, + 'DBIx substitution invokes nonempty omniholder branch once', +); + +done_testing; From 109564fa4cba16b517530a00e6ceab4da6a209a2 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:32 +0200 Subject: [PATCH 25/60] fix(regex): close combined corpus regressions (#1030) Squashed integration of PR #1030 at 5f2ce0f7d. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 79 +- dev/tools/generate_joni_line_break_data.pl | 233 ++ ...rate_perl_unicode_general_category_data.pl | 226 ++ docs/reference/feature-matrix.md | 7 +- .../regex/PerlUnicodeGeneralCategoryData.java | 2628 +++++++++++++++++ .../runtime/regex/UnicodeResolver.java | 64 +- .../PerlUnicodeGeneralCategoryDataTest.java | 25 + .../unit/regex/line_boundary_rules.t | 71 + .../regex/unicode_binary_property_values.t | 16 + .../regex/unicode_general_category_pinned.t | 25 + .../joni/src/org/joni/ArrayCompiler.java | 8 + .../joni/src/org/joni/ByteCodeMachine.java | 339 +++ third_party/joni/src/org/joni/Lexer.java | 6 + .../joni/src/org/joni/LineBreakData.java | 568 ++++ .../joni/constants/internal/AnchorType.java | 6 + .../org/joni/constants/internal/OPCode.java | 6 + .../org/joni/test/TestPerlLineBoundary.java | 102 + 17 files changed, 4379 insertions(+), 30 deletions(-) create mode 100644 dev/tools/generate_joni_line_break_data.pl create mode 100644 dev/tools/generate_perl_unicode_general_category_data.pl create mode 100644 src/main/java/org/perlonjava/runtime/regex/PerlUnicodeGeneralCategoryData.java create mode 100644 src/test/java/org/perlonjava/runtime/regex/PerlUnicodeGeneralCategoryDataTest.java create mode 100644 src/test/resources/unit/regex/line_boundary_rules.t create mode 100644 src/test/resources/unit/regex/unicode_binary_property_values.t create mode 100644 src/test/resources/unit/regex/unicode_general_category_pinned.t create mode 100644 third_party/joni/src/org/joni/LineBreakData.java create mode 100644 third_party/joni/test/org/joni/test/TestPerlLineBoundary.java diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 64241b4d39..a02f5c3578 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,16 +224,17 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1028, stacked +The published integration stack is preserved through draft PR #1029, stacked on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the first 524 lines of retired Java-only preprocessor code. Every published slice has a warning-free combined `make` checkpoint. PR #1027 adds native sentence boundaries; PR #1028 adds independently validated alpha assertion aliases and -native word boundaries. The current WIP integrates corrected global zero-width -`/g` progression and pinned Perl 5.44 Unicode 17.0 Age properties while line -boundaries remain a parallel integration slice. +native word boundaries. PR #1029 integrates corrected global zero-width `/g` +progression and pinned Perl 5.44 Unicode 17.0 Age properties. The new WIP is +closing combined-corpus property regressions, adding pinned General_Category +sets, and integrating exact native line boundaries. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -281,6 +282,23 @@ Stable chunk 01 validation improves from 30,194 to 30,705 passing assertions identically on JVM and interpreter, with no numbered regression. This raises current generated evidence by 511 to 133,089/407,367. +`General_Category`/`gc`/`Category` assignments now resolve all atomic and +aggregate values from repository-pinned Perl 5.44 Unicode 17.0 data. Short, +long, `Is_`, colon, wildcard, and loose value aliases are generated +reproducibly without the host ICU category table. The focused oracle passes +18/18 on system Perl, JVM, and interpreter; Age remains 14/14 and invalid +property diagnostics remain 39/39. Chunk 01 improves by another 606 assertions +to 31,311/41,843 identically on both execution backends with no numbered +regression, raising property-plus-completed-sentence/word evidence to +133,695/407,367 before line integration. + +Native Joni line assertions now implement Unicode 17 UAX #14 from pinned Perl +5.44 Line_Break, General_Category, East_Asian_Width, and emoji data. The +84-assertion focused oracle passes on both execution backends and chunks 06–09 +pass 205,380/205,380 each on JVM and interpreter. Protected sentence and word +chunks remain exact, making the complete generated boundary corpus +239,866/239,866 and current generated evidence 339,075/407,367. + Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while ordinary and substitution callbacks execute, without misclassifying escaped or @@ -340,6 +358,19 @@ complete in `dev/design/phase36-regex-differential-20260817.md`; Phase 1's exit criterion is not met because Joni loses Java-passing assertions and introduces matcher-specific timeouts on both execution backends. +The post-PR-#1028 plus `/g` combined forced-Joni refresh executes all 80 files +at 74,603/331,826 on JVM and 74,607/331,826 on interpreter. Four generated +property chunks time out after producing partial TAP and require the narrow +600-second rerun; chunks 05 and 10 are exact while chunks 06–09 expose only the +assigned line-boundary gap. Six regressions versus the preceding Joni result +reduced to two fatal roots. Binary `ASCII_Hex_Digit=True` routing is now closed: +the focused Perl boolean-value oracle passes 16/16 on system Perl, JVM, and +interpreter, and `pat.t` is restored from its zero-TAP abort to the independently +tracked test-239 runtime-eval gate. PerlOnJava3 owns the remaining unbraced byte +escape `too short multibyte code string` root in `pat_advanced{,_thr}.t` and +`pat_rt_report{,_thr}.t` as part of the overlapping native Joni numeric-escape +parser slice. + ### Completed Phases - [x] Phase 0: Reproducible differential baseline (2026-08-17) @@ -381,6 +412,9 @@ matcher-specific timeouts on both execution backends. first alternative (`0703725c8`, integrated as `402102446`). The focused oracle passes 23/23, the raw omniholder reducer improves from 7/10 to 10/10 in all six Java/Joni × JVM/interpreter modes, and DBIx::Simple remains 69/69. + - [x] Reran the combined forced-Joni 80-file corpus on JVM and interpreter, + published the complete file-by-file comparison, and reduced its six actual + regressions to two fatal roots with narrow owners and rerun gates. - [x] Phase 2: Conditions and backtracking-visible state (2026-08-17) - [x] Implemented executable callback conditions, control verbs including `(*MARK:NAME)`, and callback-visible recursive capture state in Joni. @@ -489,9 +523,16 @@ matcher-specific timeouts on both execution backends. Perl 5.44 Unicode 17.0 data, including loose version, wildcard, and unassigned aliases. The focused oracle passes 14/14 and chunk 01 gains 511 assertions with no numbered regression. - - [ ] Implement the native Joni line boundary algorithm, then - close the remaining property and boundary failures before marking Phase 3 - complete. + - [x] Routed Perl boolean values for the built-in `ASCII_Hex_Digit`/`AHex` + property through the frontend set resolver. All eight true/false aliases + pass 16/16 on system Perl, JVM, and interpreter, restoring `pat.t` startup. + - [x] Generated pinned Unicode 17.0 General_Category atomic and aggregate + sets with Perl property/value aliases. The focused oracle passes 18/18 and + chunk 01 gains 606 assertions on both backends with zero regressions. + - [x] Implemented native Joni line assertions from reproducible pinned Unicode + 17.0 data. The focused oracle passes 84/84 and chunks 06–09 pass + 205,380/205,380 on both execution backends while sentence/word stay exact. + - [ ] Close the remaining property failures before marking Phase 3 complete. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) - [x] Preserved mixed executable-source provenance, nested dynamic callback @@ -533,23 +574,21 @@ matcher-specific timeouts on both execution backends. ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1028. Publish the current - post-#1028 WIP containing the completed zero-width `/g` adapter integration, - then rerun the combined forced-backend differential. -2. Continue boundary semantics natively in Joni with line breaks - (0/205,380 in chunks 06–09). PerlOnJava4 owns the non-overlapping line slice; - word boundaries are complete at 19,510/19,510 in chunk 10. Use pinned Perl 5.44 Unicode data and - the generated chunks as the release oracle; remove each simplified Java - boundary rewrite only after native parity. +1. Land review-ready PR #1024 and draft PRs #1025–#1029. Integrate native Joni + unbraced-byte and underscored numeric escape parsing, then rerun the affected + direct/thread files and confirm the last combined-corpus fatal root is closed. +2. Preserve the now-complete native Joni boundary corpus at 239,866/239,866: + sentence chunk 05, line chunks 06–09, and word chunk 10 must remain exact on + JVM and interpreter while property and parser work continues. 3. Implement the remaining property clusters in measured order: Block, - Script/Script_Extensions, Numeric_Value, Joining_Group, General_Category, + Script/Script_Extensions, Numeric_Value, Joining_Group, break-property values. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. Close underscored numeric escapes with focused standard-Perl gates. -4. Rerun the forced-Joni 80-file corpus on JVM and interpreter from the combined - head. Save complete JSON and logs, publish the missing differential report, - and compare every file with both the Phase 0 result and PR 958 under the - no-regression gate. +4. Rerun generated property chunks 01–04 on both backends with the classified + 600-second bound and retain complete TAP/JSON. After the two fatal roots and + native line boundaries integrate, refresh the complete forced-Joni 80-file + corpus and apply the no-regression gate against Phase 0 and PR 958. 5. Audit every `RegexPreprocessor` rule against the final ownership boundary. Move matcher semantics into Joni, retain only source-policy scanning, delete Java-only rewrites and compiled-pattern variants, and remove the temporary diff --git a/dev/tools/generate_joni_line_break_data.pl b/dev/tools/generate_joni_line_break_data.pl new file mode 100644 index 0000000000..ef79dd1022 --- /dev/null +++ b/dev/tools/generate_joni_line_break_data.pl @@ -0,0 +1,233 @@ +#!/usr/bin/env perl +use strict; +use warnings; +use Digest::SHA qw(sha256_hex); +use File::Spec; +use FindBin; + +my $expected_version = '17.0.0'; +my $unicore = File::Spec->catdir($FindBin::Bin, '..', '..', 'perl5', 'lib', 'unicore'); +my %sources = ( + Line_Break => [File::Spec->catfile($unicore, 'LineBreak.txt'), + 'e6a18fa91f8f6a6f8e534b1d3f128c21ada45bfe152eb6b1bcc5e15fd8ac92e6'], + General_Category => [File::Spec->catfile($unicore, 'UnicodeData.txt'), + '2e1efc1dcb59c575eedf5ccae60f95229f706ee6d031835247d843c11d96470c'], + East_Asian_Width => [File::Spec->catfile($unicore, 'EastAsianWidth.txt'), + 'ea7ce50f3444a050333448dffef1cadd9325af55cbb764b4a2280faf52170a33'], + Extended_Pictographic => [File::Spec->catfile($unicore, 'emoji', 'emoji.txt'), + '2cb2bb9455cda83e8481541ecf5b6dfda66a3bb89efa3fa7c5297eccf607b72b'], +); + +sub source_text { + my ($property) = @_; + my ($path, $expected_hash) = @{$sources{$property}}; + open my $input, '<:raw', $path or die "Cannot read $path: $!\n"; + local $/; + my $text = <$input>; + close $input or die "Cannot close $path: $!\n"; + my $actual_hash = sha256_hex($text); + die "$path SHA-256 mismatch: expected $expected_hash, found $actual_hash\n" + unless $actual_hash eq $expected_hash; + return ($path, $text); +} + +sub range_from_text { + my ($range) = @_; + my ($first, $last) = split /\.\./, $range; + return (hex($first), hex(defined $last ? $last : $first)); +} + +sub parse_property_file { + my ($property, $version_pattern, $wanted_value, $default) = @_; + my ($path, $text) = source_text($property); + die "$path is not pinned Unicode $expected_version data\n" + unless $text =~ $version_pattern; + my (@ranges, @missing); + for my $line (split /\n/, $text) { + if ($line =~ /^#\s*\@missing:\s*([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z0-9_]+)/) { + my ($range, $value) = ($1, $2); + my ($first, $last) = range_from_text($range); + push @missing, [$first, $last, $value]; + next; + } + next unless $line =~ /^([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z0-9_]+)/; + my ($range, $value) = ($1, $2); + next if defined $wanted_value && $value ne $wanted_value; + my ($first, $last) = range_from_text($range); + push @ranges, [$first, $last, defined $wanted_value ? 'Y' : $value]; + } + return {ranges => \@ranges, missing => \@missing, default => $default}; +} + +sub parse_general_category { + my ($path, $text) = source_text('General_Category'); + my (@ranges, $pending); + for my $line (split /\n/, $text) { + my @fields = split /;/, $line, -1; + next unless @fields >= 3; + my ($code, $name, $category) = (hex($fields[0]), $fields[1], $fields[2]); + if ($name =~ /, First>$/) { + $pending = [$code, $category]; + } elsif ($name =~ /, Last>$/) { + die "Unmatched UnicodeData Last range at $fields[0]\n" + unless $pending && $pending->[1] eq $category; + push @ranges, [$pending->[0], $code, $category]; + undef $pending; + } else { + push @ranges, [$code, $code, $category]; + } + } + die "Unclosed UnicodeData First range\n" if $pending; + return {ranges => \@ranges, missing => [], default => 'Cn'}; +} + +my @classes = qw( + AI AK AL AP AS B2 BA BB BK CB CJ CL CM CP CR EB EM EX GL H2 H3 HH HL HY + ID IN IS JL JT JV LF NL NS NU OP PO PR QU RI SA SG SP SY VF VI WJ XX ZW ZWJ +); +my %class_id; +@class_id{@classes} = (0 .. $#classes); + +my %maps = ( + Line_Break => parse_property_file('Line_Break', qr/^# LineBreak-\Q$expected_version\E\.txt/m, undef, 'XX'), + General_Category => parse_general_category(), + East_Asian_Width => parse_property_file('East_Asian_Width', qr/^# EastAsianWidth-\Q$expected_version\E\.txt/m, undef, 'N'), + Extended_Pictographic => parse_property_file('Extended_Pictographic', qr/^# Version: 17\.0$/m, + 'Extended_Pictographic', 'N'), +); + +my %starts = (0 => 1); +for my $map (values %maps) { + for my $range (@{$map->{ranges}}, @{$map->{missing}}) { + $starts{$range->[0]} = 1; + $starts{$range->[1] + 1} = 1 if $range->[1] < 0x10ffff; + } +} +my @starts = sort { $a <=> $b } keys %starts; + +sub value_at { + my ($map, $code) = @_; + my $ranges = $map->{ranges}; + my ($low, $high) = (0, $#$ranges); + while ($low <= $high) { + my $middle = ($low + $high) >> 1; + if ($ranges->[$middle][0] <= $code) { + $low = $middle + 1; + } else { + $high = $middle - 1; + } + } + return $ranges->[$high][2] if $high >= 0 && $code <= $ranges->[$high][1]; + my $value = $map->{default}; + for my $range (@{$map->{missing}}) { + $value = $range->[2] if $range->[0] <= $code && $code <= $range->[1]; + } + return $value; +} + +my @values; +for my $code (@starts) { + my $lb = value_at($maps{Line_Break}, $code); + my $gc = value_at($maps{General_Category}, $code); + my $ea = value_at($maps{East_Asian_Width}, $code); + my $ep = value_at($maps{Extended_Pictographic}, $code); + + $lb = 'AL' if $lb eq 'AI' || $lb eq 'SG' || $lb eq 'XX'; + $lb = $gc eq 'Mn' || $gc eq 'Mc' ? 'CM' : 'AL' if $lb eq 'SA'; + $lb = 'NS' if $lb eq 'CJ'; + die "Unknown Line_Break value '$lb'\n" unless exists $class_id{$lb}; + + my $packed = $class_id{$lb}; + $packed |= 1 << 6 if $gc eq 'Pi'; + $packed |= 1 << 7 if $gc eq 'Pf'; + $packed |= 1 << 8 if $ea eq 'F' || $ea eq 'W' || $ea eq 'H'; + $packed |= 1 << 9 if $ep eq 'Y' && $gc eq 'Cn'; + push @values, $packed; +} + +# Coalesce adjacent union intervals whose packed value is identical. +my (@coalesced_starts, @coalesced_values); +for my $i (0 .. $#starts) { + next if @coalesced_values && $values[$i] == $coalesced_values[-1]; + push @coalesced_starts, $starts[$i]; + push @coalesced_values, $values[$i]; +} + +print <<'HEADER'; +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni; + +// Generated by dev/tools/generate_joni_line_break_data.pl. Do not edit manually. +final class LineBreakData { +HEADER + +print " static final String UNICODE_VERSION = \"$expected_version\";\n"; +for my $class (@classes) { + print " static final short $class = $class_id{$class};\n"; +} +print <<'FLAGS'; + static final short CLASS_MASK = 0x3f; + static final short INITIAL_PUNCTUATION = 1 << 6; + static final short FINAL_PUNCTUATION = 1 << 7; + static final short EAST_ASIAN = 1 << 8; + static final short UNASSIGNED_EXTENDED_PICTOGRAPHIC = 1 << 9; + +FLAGS + +print " private static final int[] STARTS = {\n"; +for (my $i = 0; $i < @coalesced_starts; $i += 10) { + my $end = $i + 9 < $#coalesced_starts ? $i + 9 : $#coalesced_starts; + print " ", join(', ', map { sprintf '0x%X', $coalesced_starts[$_] } $i .. $end), ",\n"; +} +print " };\n\n private static final short[] VALUES = {\n"; +for (my $i = 0; $i < @coalesced_values; $i += 16) { + my $end = $i + 15 < $#coalesced_values ? $i + 15 : $#coalesced_values; + print " ", join(', ', @coalesced_values[$i .. $end]), ",\n"; +} +print <<'FOOTER'; + }; + + static short propertyOf(int codePoint) { + int low = 0; + int high = STARTS.length - 1; + while (low <= high) { + int middle = (low + high) >>> 1; + if (STARTS[middle] <= codePoint) { + low = middle + 1; + } else { + high = middle - 1; + } + } + return VALUES[Math.max(0, high)]; + } + + static short lineClass(short property) { + return (short)(property & CLASS_MASK); + } + + static boolean has(short property, short flag) { + return (property & flag) != 0; + } + + private LineBreakData() { + } +} +FOOTER diff --git a/dev/tools/generate_perl_unicode_general_category_data.pl b/dev/tools/generate_perl_unicode_general_category_data.pl new file mode 100644 index 0000000000..1de28460ae --- /dev/null +++ b/dev/tools/generate_perl_unicode_general_category_data.pl @@ -0,0 +1,226 @@ +#!/usr/bin/env perl +use strict; +use warnings; +use Digest::SHA qw(sha256_hex); + +my $unicode_root = 'perl5/lib/unicore'; +my $expected_unicode_version = '17.0.0'; +my %expected_hash = ( + 'extracted/DGeneralCategory.txt' => + 'd62e5bab70ca74f099343f71224fa051cb1fdd61a1ab45c0488c44cfc0b6102e', + 'PropValueAliases.txt' => + '670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01', +); + +open my $version_fh, '<', "$unicode_root/version" + or die "Can't read Unicode version: $!\n"; +chomp(my $unicode_version = <$version_fh>); +close $version_fh; +die "Expected Unicode $expected_unicode_version, found $unicode_version\n" + unless $unicode_version eq $expected_unicode_version; + +sub read_pinned_file { + my ($relative) = @_; + my $path = "$unicode_root/$relative"; + open my $fh, '<:raw', $path or die "Can't read $path: $!\n"; + local $/; + my $text = <$fh>; + close $fh; + my $actual = sha256_hex($text); + die "$path SHA-256 mismatch: expected $expected_hash{$relative}, found $actual\n" + unless $actual eq $expected_hash{$relative}; + return $text; +} + +sub loose_name { + my ($name) = @_; + $name = lc $name; + $name =~ s/[\s_-]//g; + return $name; +} + +my $category_text = read_pinned_file('extracted/DGeneralCategory.txt'); +die "General_Category data is not Unicode $expected_unicode_version\n" + unless $category_text =~ /^# DerivedGeneralCategory-\Q$expected_unicode_version\E\.txt/m; + +my (@values, %value_index, @ranges); +for my $line (split /\n/, $category_text) { + next unless $line =~ /^([0-9A-F]+)(?:\.\.([0-9A-F]+))?\s*;\s*([A-Za-z]+)/; + my ($start, $end, $value) = + (hex($1), defined($2) ? hex($2) : hex($1), $3); + if (!exists $value_index{$value}) { + $value_index{$value} = scalar @values; + push @values, $value; + } + my $index = $value_index{$value}; + if (@ranges && $ranges[-1][2] == $index && $ranges[-1][1] + 1 == $start) { + $ranges[-1][1] = $end; + } else { + push @ranges, [$start, $end, $index]; + } +} +die "No General_Category ranges found\n" unless @ranges; + +# DerivedGeneralCategory contains the 30 atomic values. Perl also exposes the +# Unicode aggregate aliases from PropertyValueAliases.txt; materialize their +# unions from the same pinned atomic ranges so no host Unicode table is used. +my %composite = ( + C => [qw(Cc Cf Cn Co Cs)], + L => [qw(Ll Lm Lo Lt Lu)], + LC => [qw(Ll Lt Lu)], + M => [qw(Mc Me Mn)], + N => [qw(Nd Nl No)], + P => [qw(Pc Pd Pe Pf Pi Po Ps)], + S => [qw(Sc Sk Sm So)], + Z => [qw(Zl Zp Zs)], +); +my @atomic_ranges = @ranges; +for my $value (qw(C L LC M N P S Z)) { + $value_index{$value} = scalar @values; + push @values, $value; + my %member = map { + die "Composite $value references unknown category $_\n" + unless exists $value_index{$_}; + $value_index{$_} => 1; + } @{$composite{$value}}; + push @ranges, map { + [$_->[0], $_->[1], $value_index{$value}] + } grep { $member{$_->[2]} } @atomic_ranges; +} + +my $alias_text = read_pinned_file('PropValueAliases.txt'); +die "Property value aliases are not Unicode $expected_unicode_version\n" + unless $alias_text =~ /^# PropertyValueAliases-\Q$expected_unicode_version\E\.txt/m; + +my %aliases; +for my $line (split /\n/, $alias_text) { + $line =~ s/\s*#.*$//; + next unless $line =~ /\S/; + my @fields = map { + my $field = $_; + $field =~ s/^\s+|\s+$//g; + $field; + } split /;/, $line; + next unless @fields >= 3 && lc($fields[0]) eq 'gc'; + my $canonical = $fields[1]; + die "General_Category alias has unknown value $canonical\n" + unless exists $value_index{$canonical}; + for my $alias (@fields[1 .. $#fields]) { + next unless length $alias; + my $loose = loose_name($alias); + if (exists $aliases{$loose} && $aliases{$loose} ne $canonical) { + die "Conflicting General_Category alias $alias\n"; + } + $aliases{$loose} = $canonical; + } +} +for my $value (@values) { + $aliases{loose_name($value)} //= $value; +} + +print <<'HEADER'; +/* + * Generated from Perl 5.44's Unicode Character Database. Do not edit manually. + * + * Unicode data source copyright: + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in + * the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + */ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; + +import java.util.HashMap; +import java.util.Map; + +final class PerlUnicodeGeneralCategoryData { +HEADER + +print " static final String UNICODE_VERSION = \"$unicode_version\";\n"; +print " private static final String[] VALUES = {\n "; +print join(', ', map { qq{"$_"} } @values); +print "\n };\n\n"; +my $range_chunk_size = 400; +my $range_chunk_count = int((@ranges + $range_chunk_size - 1) / $range_chunk_size); +for my $chunk (0 .. $range_chunk_count - 1) { + my $first = $chunk * $range_chunk_size; + my $last = $first + $range_chunk_size - 1; + $last = $#ranges if $last > $#ranges; + print " private static int[] rangeChunk$chunk() {\n"; + print " return new int[] {\n"; + for (my $i = $first; $i <= $last; $i += 4) { + my $end = $i + 3 < $last ? $i + 3 : $last; + print " ", join(', ', map { + sprintf '0x%X, 0x%X, %d', @{$ranges[$_]}[0, 1, 2] + } $i .. $end), ",\n"; + } + print " };\n"; + print " }\n\n"; +} +print " private static final int[][] RANGE_CHUNKS = {\n "; +print join(', ', map { "rangeChunk$_()" } 0 .. $range_chunk_count - 1); +print "\n };\n\n"; +print " private static final String[] ALIASES = {\n"; +my @alias_names = sort keys %aliases; +for (my $i = 0; $i < @alias_names; $i += 4) { + my $end = $i + 3 < $#alias_names ? $i + 3 : $#alias_names; + print " ", join(', ', map { + my $name = $alias_names[$_]; + qq{"$name", "} . $aliases{$name} . qq{"} + } $i .. $end), ",\n"; +} +print <<'FOOTER'; + }; + + private static final UnicodeSet[] SETS = buildSets(); + private static final Map ALIAS_INDEX = buildAliasIndex(); + + static UnicodeSet resolve(String value) { + Integer index = ALIAS_INDEX.get(normalizeValue(value)); + return index == null ? null : SETS[index]; + } + + private static UnicodeSet[] buildSets() { + UnicodeSet[] sets = new UnicodeSet[VALUES.length]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + for (int[] ranges : RANGE_CHUNKS) { + for (int i = 0; i < ranges.length; i += 3) { + sets[ranges[i + 2]].add(ranges[i], ranges[i + 1]); + } + } + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + + private static Map buildAliasIndex() { + Map valueIndexes = new HashMap<>(); + for (int i = 0; i < VALUES.length; i++) valueIndexes.put(VALUES[i], i); + Map indexes = new HashMap<>(); + for (int i = 0; i < ALIASES.length; i += 2) { + indexes.put(ALIASES[i], valueIndexes.get(ALIASES[i + 1])); + } + return Map.copyOf(indexes); + } + + private static String normalizeValue(String value) { + String normalized = value.trim(); + if (normalized.startsWith(":\\A") && normalized.endsWith("\\z:") + && normalized.length() > 6) { + normalized = normalized.substring(3, normalized.length() - 3); + } + StringBuilder loose = new StringBuilder(normalized.length()); + for (int i = 0; i < normalized.length(); i++) { + char ch = normalized.charAt(i); + if (!Character.isWhitespace(ch) && ch != '-' && ch != '_') { + loose.append(Character.toLowerCase(ch)); + } + } + return loose.toString(); + } + + private PerlUnicodeGeneralCategoryData() { + } +} +FOOTER diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index 6a5999e0d5..b912639f82 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -371,14 +371,14 @@ my @copy = @{$z}; # ERROR - ✅ **Inline comments**: `(?#comment)` in regex is implemented. - ✅ **caret modifier**: `(?^` embedded pattern-match modifier, shorthand equivalent to "d-imnsx". - ✅ **\b inside character class**: `[\b]` is supported in regex. -- 🟡 **Unicode boundary assertions**: `\b{gcb}`, `\b{sb}`, and `\b{wb}` (and their `\B` forms) execute natively in Joni with complete authoritative generated sections on both backends. Line boundaries still use a simplified frontend rewrite pending native `\b{lb}` integration. +- ✅ **Unicode boundary assertions**: `\b{gcb}`, `\b{sb}`, `\b{wb}`, and `\b{lb}` (and their `\B` forms) execute natively in Joni from reproducible pinned Perl 5.44 Unicode 17.0 data. The complete 239,866-assertion generated boundary corpus passes on both backends. - ✅ **Variable Interpolation in Regex**: Features like `${var}` for embedding variables. - ✅ **Non-capturing groups**: `(?:...)` is implemented. - ✅ **Named Capture Groups**: Defining named capture groups using `(?...)` or `(?'name'...)` is supported. - ✅ **Backreferences to Named Groups**: Using `\k` or `\g{name}` for backreferences to named groups is supported. - ✅ **Relative Backreferences**: Using `\g{-n}` for relative backreferences. - ✅ **Basic Unicode Properties**: Common `\p{...}` and `\P{...}` forms such as `\p{L}` execute through Joni. -- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age` and cumulative `In`/`Present_In` are generated from pinned Perl 5.44 Unicode 17.0 data, including loose versions and `Unassigned`/`NA`; other generated property/value aliases still have pinned acceptance/rejection gaps. +- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, and all General_Category assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. - ✅ **Possessive Quantifiers**: Quantifiers like `*+`, `++`, `?+`, and `{n,m}+`, which disable backtracking, are supported. - ✅ **Atomic Grouping**: Use of `(?>...)` for atomic groups is supported. - ✅ **`\K` assertion**: Keep left — in `s///`, text before `\K` is preserved; match variables reflect only the portion after `\K`. @@ -397,10 +397,11 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 133,089/407,367: chunks 01–04 expose the remaining property/value alias gaps, and the remaining boundary failures are line breaks. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 339,075/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. - ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. - ✅ **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter. - ✅ **Unicode Word Boundaries**: Native `\b{wb}`/`\B{wb}` implement WB1–WB16 and WB999 from reproducible Perl 5.44 Unicode 17.0 word-break and extended-pictographic tables. Authoritative chunk 10 passes 19,510/19,510 identically on JVM and interpreter. +- ✅ **Unicode Line Boundaries**: Native `\b{lb}`/`\B{lb}` implement Unicode 17 UAX #14 from reproducible pinned line-break, category, width, and emoji tables. Authoritative chunks 06–09 pass 205,380/205,380 identically on JVM and interpreter. - ✅ **Embedded Code in Regex**: `(?{ code })`, optimistic callbacks `(*{ code })`, executable callback conditions, and `(??{ code })` run as lexical closures in Joni with provisional captures and backtracking unwind. Callback `local` frames follow matcher paths, and escaped loop control or `goto` stops at the callback pseudo-block boundary. `$^N` follows capture-close order independently of `$+`. - ✅ **Regex Debugging**: Lexically scoped `use/no re 'debug'` and `debugcolor` are supported, including runtime snapshot ownership. - ✅ **Runtime Regex Evaluation**: `use re 'eval'` controls whether interpolated patterns containing eval groups may compile. Admitted runtime source is compiled into lexical callback closures and preserves its package, visible lexical cells, Unicode or byte source type, default regex modifiers, and match-once state. Literal callbacks, interpolated `qr//` values (including local, referenced, and tied arrays), raw runtime eval groups, callback conditions, and standalone `(?(DEFINE)...)` containers may be composed in one Joni pattern; dynamic callbacks may return further admitted executable source. diff --git a/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeGeneralCategoryData.java b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeGeneralCategoryData.java new file mode 100644 index 0000000000..6a28269abb --- /dev/null +++ b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeGeneralCategoryData.java @@ -0,0 +1,2628 @@ +/* + * Generated from Perl 5.44's Unicode Character Database. Do not edit manually. + * + * Unicode data source copyright: + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in + * the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + */ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; + +import java.util.HashMap; +import java.util.Map; + +final class PerlUnicodeGeneralCategoryData { + static final String UNICODE_VERSION = "17.0.0"; + private static final String[] VALUES = { + "Cn", "Lu", "Ll", "Lt", "Lm", "Lo", "Mn", "Me", "Mc", "Nd", "Nl", "No", "Zs", "Zl", "Zp", "Cc", "Cf", "Co", "Cs", "Pd", "Ps", "Pe", "Pc", "Po", "Sm", "Sc", "Sk", "So", "Pi", "Pf", "C", "L", "LC", "M", "N", "P", "S", "Z" + }; + + private static int[] rangeChunk0() { + return new int[] { + 0x378, 0x379, 0, 0x380, 0x383, 0, 0x38B, 0x38B, 0, 0x38D, 0x38D, 0, + 0x3A2, 0x3A2, 0, 0x530, 0x530, 0, 0x557, 0x558, 0, 0x58B, 0x58C, 0, + 0x590, 0x590, 0, 0x5C8, 0x5CF, 0, 0x5EB, 0x5EE, 0, 0x5F5, 0x5FF, 0, + 0x70E, 0x70E, 0, 0x74B, 0x74C, 0, 0x7B2, 0x7BF, 0, 0x7FB, 0x7FC, 0, + 0x82E, 0x82F, 0, 0x83F, 0x83F, 0, 0x85C, 0x85D, 0, 0x85F, 0x85F, 0, + 0x86B, 0x86F, 0, 0x892, 0x896, 0, 0x984, 0x984, 0, 0x98D, 0x98E, 0, + 0x991, 0x992, 0, 0x9A9, 0x9A9, 0, 0x9B1, 0x9B1, 0, 0x9B3, 0x9B5, 0, + 0x9BA, 0x9BB, 0, 0x9C5, 0x9C6, 0, 0x9C9, 0x9CA, 0, 0x9CF, 0x9D6, 0, + 0x9D8, 0x9DB, 0, 0x9DE, 0x9DE, 0, 0x9E4, 0x9E5, 0, 0x9FF, 0xA00, 0, + 0xA04, 0xA04, 0, 0xA0B, 0xA0E, 0, 0xA11, 0xA12, 0, 0xA29, 0xA29, 0, + 0xA31, 0xA31, 0, 0xA34, 0xA34, 0, 0xA37, 0xA37, 0, 0xA3A, 0xA3B, 0, + 0xA3D, 0xA3D, 0, 0xA43, 0xA46, 0, 0xA49, 0xA4A, 0, 0xA4E, 0xA50, 0, + 0xA52, 0xA58, 0, 0xA5D, 0xA5D, 0, 0xA5F, 0xA65, 0, 0xA77, 0xA80, 0, + 0xA84, 0xA84, 0, 0xA8E, 0xA8E, 0, 0xA92, 0xA92, 0, 0xAA9, 0xAA9, 0, + 0xAB1, 0xAB1, 0, 0xAB4, 0xAB4, 0, 0xABA, 0xABB, 0, 0xAC6, 0xAC6, 0, + 0xACA, 0xACA, 0, 0xACE, 0xACF, 0, 0xAD1, 0xADF, 0, 0xAE4, 0xAE5, 0, + 0xAF2, 0xAF8, 0, 0xB00, 0xB00, 0, 0xB04, 0xB04, 0, 0xB0D, 0xB0E, 0, + 0xB11, 0xB12, 0, 0xB29, 0xB29, 0, 0xB31, 0xB31, 0, 0xB34, 0xB34, 0, + 0xB3A, 0xB3B, 0, 0xB45, 0xB46, 0, 0xB49, 0xB4A, 0, 0xB4E, 0xB54, 0, + 0xB58, 0xB5B, 0, 0xB5E, 0xB5E, 0, 0xB64, 0xB65, 0, 0xB78, 0xB81, 0, + 0xB84, 0xB84, 0, 0xB8B, 0xB8D, 0, 0xB91, 0xB91, 0, 0xB96, 0xB98, 0, + 0xB9B, 0xB9B, 0, 0xB9D, 0xB9D, 0, 0xBA0, 0xBA2, 0, 0xBA5, 0xBA7, 0, + 0xBAB, 0xBAD, 0, 0xBBA, 0xBBD, 0, 0xBC3, 0xBC5, 0, 0xBC9, 0xBC9, 0, + 0xBCE, 0xBCF, 0, 0xBD1, 0xBD6, 0, 0xBD8, 0xBE5, 0, 0xBFB, 0xBFF, 0, + 0xC0D, 0xC0D, 0, 0xC11, 0xC11, 0, 0xC29, 0xC29, 0, 0xC3A, 0xC3B, 0, + 0xC45, 0xC45, 0, 0xC49, 0xC49, 0, 0xC4E, 0xC54, 0, 0xC57, 0xC57, 0, + 0xC5B, 0xC5B, 0, 0xC5E, 0xC5F, 0, 0xC64, 0xC65, 0, 0xC70, 0xC76, 0, + 0xC8D, 0xC8D, 0, 0xC91, 0xC91, 0, 0xCA9, 0xCA9, 0, 0xCB4, 0xCB4, 0, + 0xCBA, 0xCBB, 0, 0xCC5, 0xCC5, 0, 0xCC9, 0xCC9, 0, 0xCCE, 0xCD4, 0, + 0xCD7, 0xCDB, 0, 0xCDF, 0xCDF, 0, 0xCE4, 0xCE5, 0, 0xCF0, 0xCF0, 0, + 0xCF4, 0xCFF, 0, 0xD0D, 0xD0D, 0, 0xD11, 0xD11, 0, 0xD45, 0xD45, 0, + 0xD49, 0xD49, 0, 0xD50, 0xD53, 0, 0xD64, 0xD65, 0, 0xD80, 0xD80, 0, + 0xD84, 0xD84, 0, 0xD97, 0xD99, 0, 0xDB2, 0xDB2, 0, 0xDBC, 0xDBC, 0, + 0xDBE, 0xDBF, 0, 0xDC7, 0xDC9, 0, 0xDCB, 0xDCE, 0, 0xDD5, 0xDD5, 0, + 0xDD7, 0xDD7, 0, 0xDE0, 0xDE5, 0, 0xDF0, 0xDF1, 0, 0xDF5, 0xE00, 0, + 0xE3B, 0xE3E, 0, 0xE5C, 0xE80, 0, 0xE83, 0xE83, 0, 0xE85, 0xE85, 0, + 0xE8B, 0xE8B, 0, 0xEA4, 0xEA4, 0, 0xEA6, 0xEA6, 0, 0xEBE, 0xEBF, 0, + 0xEC5, 0xEC5, 0, 0xEC7, 0xEC7, 0, 0xECF, 0xECF, 0, 0xEDA, 0xEDB, 0, + 0xEE0, 0xEFF, 0, 0xF48, 0xF48, 0, 0xF6D, 0xF70, 0, 0xF98, 0xF98, 0, + 0xFBD, 0xFBD, 0, 0xFCD, 0xFCD, 0, 0xFDB, 0xFFF, 0, 0x10C6, 0x10C6, 0, + 0x10C8, 0x10CC, 0, 0x10CE, 0x10CF, 0, 0x1249, 0x1249, 0, 0x124E, 0x124F, 0, + 0x1257, 0x1257, 0, 0x1259, 0x1259, 0, 0x125E, 0x125F, 0, 0x1289, 0x1289, 0, + 0x128E, 0x128F, 0, 0x12B1, 0x12B1, 0, 0x12B6, 0x12B7, 0, 0x12BF, 0x12BF, 0, + 0x12C1, 0x12C1, 0, 0x12C6, 0x12C7, 0, 0x12D7, 0x12D7, 0, 0x1311, 0x1311, 0, + 0x1316, 0x1317, 0, 0x135B, 0x135C, 0, 0x137D, 0x137F, 0, 0x139A, 0x139F, 0, + 0x13F6, 0x13F7, 0, 0x13FE, 0x13FF, 0, 0x169D, 0x169F, 0, 0x16F9, 0x16FF, 0, + 0x1716, 0x171E, 0, 0x1737, 0x173F, 0, 0x1754, 0x175F, 0, 0x176D, 0x176D, 0, + 0x1771, 0x1771, 0, 0x1774, 0x177F, 0, 0x17DE, 0x17DF, 0, 0x17EA, 0x17EF, 0, + 0x17FA, 0x17FF, 0, 0x181A, 0x181F, 0, 0x1879, 0x187F, 0, 0x18AB, 0x18AF, 0, + 0x18F6, 0x18FF, 0, 0x191F, 0x191F, 0, 0x192C, 0x192F, 0, 0x193C, 0x193F, 0, + 0x1941, 0x1943, 0, 0x196E, 0x196F, 0, 0x1975, 0x197F, 0, 0x19AC, 0x19AF, 0, + 0x19CA, 0x19CF, 0, 0x19DB, 0x19DD, 0, 0x1A1C, 0x1A1D, 0, 0x1A5F, 0x1A5F, 0, + 0x1A7D, 0x1A7E, 0, 0x1A8A, 0x1A8F, 0, 0x1A9A, 0x1A9F, 0, 0x1AAE, 0x1AAF, 0, + 0x1ADE, 0x1ADF, 0, 0x1AEC, 0x1AFF, 0, 0x1B4D, 0x1B4D, 0, 0x1BF4, 0x1BFB, 0, + 0x1C38, 0x1C3A, 0, 0x1C4A, 0x1C4C, 0, 0x1C8B, 0x1C8F, 0, 0x1CBB, 0x1CBC, 0, + 0x1CC8, 0x1CCF, 0, 0x1CFB, 0x1CFF, 0, 0x1F16, 0x1F17, 0, 0x1F1E, 0x1F1F, 0, + 0x1F46, 0x1F47, 0, 0x1F4E, 0x1F4F, 0, 0x1F58, 0x1F58, 0, 0x1F5A, 0x1F5A, 0, + 0x1F5C, 0x1F5C, 0, 0x1F5E, 0x1F5E, 0, 0x1F7E, 0x1F7F, 0, 0x1FB5, 0x1FB5, 0, + 0x1FC5, 0x1FC5, 0, 0x1FD4, 0x1FD5, 0, 0x1FDC, 0x1FDC, 0, 0x1FF0, 0x1FF1, 0, + 0x1FF5, 0x1FF5, 0, 0x1FFF, 0x1FFF, 0, 0x2065, 0x2065, 0, 0x2072, 0x2073, 0, + 0x208F, 0x208F, 0, 0x209D, 0x209F, 0, 0x20C2, 0x20CF, 0, 0x20F1, 0x20FF, 0, + 0x218C, 0x218F, 0, 0x242A, 0x243F, 0, 0x244B, 0x245F, 0, 0x2B74, 0x2B75, 0, + 0x2CF4, 0x2CF8, 0, 0x2D26, 0x2D26, 0, 0x2D28, 0x2D2C, 0, 0x2D2E, 0x2D2F, 0, + 0x2D68, 0x2D6E, 0, 0x2D71, 0x2D7E, 0, 0x2D97, 0x2D9F, 0, 0x2DA7, 0x2DA7, 0, + 0x2DAF, 0x2DAF, 0, 0x2DB7, 0x2DB7, 0, 0x2DBF, 0x2DBF, 0, 0x2DC7, 0x2DC7, 0, + 0x2DCF, 0x2DCF, 0, 0x2DD7, 0x2DD7, 0, 0x2DDF, 0x2DDF, 0, 0x2E5E, 0x2E7F, 0, + 0x2E9A, 0x2E9A, 0, 0x2EF4, 0x2EFF, 0, 0x2FD6, 0x2FEF, 0, 0x3040, 0x3040, 0, + 0x3097, 0x3098, 0, 0x3100, 0x3104, 0, 0x3130, 0x3130, 0, 0x318F, 0x318F, 0, + 0x31E6, 0x31EE, 0, 0x321F, 0x321F, 0, 0xA48D, 0xA48F, 0, 0xA4C7, 0xA4CF, 0, + 0xA62C, 0xA63F, 0, 0xA6F8, 0xA6FF, 0, 0xA7DD, 0xA7F0, 0, 0xA82D, 0xA82F, 0, + 0xA83A, 0xA83F, 0, 0xA878, 0xA87F, 0, 0xA8C6, 0xA8CD, 0, 0xA8DA, 0xA8DF, 0, + 0xA954, 0xA95E, 0, 0xA97D, 0xA97F, 0, 0xA9CE, 0xA9CE, 0, 0xA9DA, 0xA9DD, 0, + 0xA9FF, 0xA9FF, 0, 0xAA37, 0xAA3F, 0, 0xAA4E, 0xAA4F, 0, 0xAA5A, 0xAA5B, 0, + 0xAAC3, 0xAADA, 0, 0xAAF7, 0xAB00, 0, 0xAB07, 0xAB08, 0, 0xAB0F, 0xAB10, 0, + 0xAB17, 0xAB1F, 0, 0xAB27, 0xAB27, 0, 0xAB2F, 0xAB2F, 0, 0xAB6C, 0xAB6F, 0, + 0xABEE, 0xABEF, 0, 0xABFA, 0xABFF, 0, 0xD7A4, 0xD7AF, 0, 0xD7C7, 0xD7CA, 0, + 0xD7FC, 0xD7FF, 0, 0xFA6E, 0xFA6F, 0, 0xFADA, 0xFAFF, 0, 0xFB07, 0xFB12, 0, + 0xFB18, 0xFB1C, 0, 0xFB37, 0xFB37, 0, 0xFB3D, 0xFB3D, 0, 0xFB3F, 0xFB3F, 0, + 0xFB42, 0xFB42, 0, 0xFB45, 0xFB45, 0, 0xFDD0, 0xFDEF, 0, 0xFE1A, 0xFE1F, 0, + 0xFE53, 0xFE53, 0, 0xFE67, 0xFE67, 0, 0xFE6C, 0xFE6F, 0, 0xFE75, 0xFE75, 0, + 0xFEFD, 0xFEFE, 0, 0xFF00, 0xFF00, 0, 0xFFBF, 0xFFC1, 0, 0xFFC8, 0xFFC9, 0, + 0xFFD0, 0xFFD1, 0, 0xFFD8, 0xFFD9, 0, 0xFFDD, 0xFFDF, 0, 0xFFE7, 0xFFE7, 0, + 0xFFEF, 0xFFF8, 0, 0xFFFE, 0xFFFF, 0, 0x1000C, 0x1000C, 0, 0x10027, 0x10027, 0, + 0x1003B, 0x1003B, 0, 0x1003E, 0x1003E, 0, 0x1004E, 0x1004F, 0, 0x1005E, 0x1007F, 0, + 0x100FB, 0x100FF, 0, 0x10103, 0x10106, 0, 0x10134, 0x10136, 0, 0x1018F, 0x1018F, 0, + 0x1019D, 0x1019F, 0, 0x101A1, 0x101CF, 0, 0x101FE, 0x1027F, 0, 0x1029D, 0x1029F, 0, + 0x102D1, 0x102DF, 0, 0x102FC, 0x102FF, 0, 0x10324, 0x1032C, 0, 0x1034B, 0x1034F, 0, + 0x1037B, 0x1037F, 0, 0x1039E, 0x1039E, 0, 0x103C4, 0x103C7, 0, 0x103D6, 0x103FF, 0, + 0x1049E, 0x1049F, 0, 0x104AA, 0x104AF, 0, 0x104D4, 0x104D7, 0, 0x104FC, 0x104FF, 0, + 0x10528, 0x1052F, 0, 0x10564, 0x1056E, 0, 0x1057B, 0x1057B, 0, 0x1058B, 0x1058B, 0, + 0x10593, 0x10593, 0, 0x10596, 0x10596, 0, 0x105A2, 0x105A2, 0, 0x105B2, 0x105B2, 0, + 0x105BA, 0x105BA, 0, 0x105BD, 0x105BF, 0, 0x105F4, 0x105FF, 0, 0x10737, 0x1073F, 0, + 0x10756, 0x1075F, 0, 0x10768, 0x1077F, 0, 0x10786, 0x10786, 0, 0x107B1, 0x107B1, 0, + 0x107BB, 0x107FF, 0, 0x10806, 0x10807, 0, 0x10809, 0x10809, 0, 0x10836, 0x10836, 0, + 0x10839, 0x1083B, 0, 0x1083D, 0x1083E, 0, 0x10856, 0x10856, 0, 0x1089F, 0x108A6, 0, + 0x108B0, 0x108DF, 0, 0x108F3, 0x108F3, 0, 0x108F6, 0x108FA, 0, 0x1091C, 0x1091E, 0, + 0x1093A, 0x1093E, 0, 0x1095A, 0x1097F, 0, 0x109B8, 0x109BB, 0, 0x109D0, 0x109D1, 0, + 0x10A04, 0x10A04, 0, 0x10A07, 0x10A0B, 0, 0x10A14, 0x10A14, 0, 0x10A18, 0x10A18, 0, + 0x10A36, 0x10A37, 0, 0x10A3B, 0x10A3E, 0, 0x10A49, 0x10A4F, 0, 0x10A59, 0x10A5F, 0, + 0x10AA0, 0x10ABF, 0, 0x10AE7, 0x10AEA, 0, 0x10AF7, 0x10AFF, 0, 0x10B36, 0x10B38, 0, + }; + } + + private static int[] rangeChunk1() { + return new int[] { + 0x10B56, 0x10B57, 0, 0x10B73, 0x10B77, 0, 0x10B92, 0x10B98, 0, 0x10B9D, 0x10BA8, 0, + 0x10BB0, 0x10BFF, 0, 0x10C49, 0x10C7F, 0, 0x10CB3, 0x10CBF, 0, 0x10CF3, 0x10CF9, 0, + 0x10D28, 0x10D2F, 0, 0x10D3A, 0x10D3F, 0, 0x10D66, 0x10D68, 0, 0x10D86, 0x10D8D, 0, + 0x10D90, 0x10E5F, 0, 0x10E7F, 0x10E7F, 0, 0x10EAA, 0x10EAA, 0, 0x10EAE, 0x10EAF, 0, + 0x10EB2, 0x10EC1, 0, 0x10EC8, 0x10ECF, 0, 0x10ED9, 0x10EF9, 0, 0x10F28, 0x10F2F, 0, + 0x10F5A, 0x10F6F, 0, 0x10F8A, 0x10FAF, 0, 0x10FCC, 0x10FDF, 0, 0x10FF7, 0x10FFF, 0, + 0x1104E, 0x11051, 0, 0x11076, 0x1107E, 0, 0x110C3, 0x110CC, 0, 0x110CE, 0x110CF, 0, + 0x110E9, 0x110EF, 0, 0x110FA, 0x110FF, 0, 0x11135, 0x11135, 0, 0x11148, 0x1114F, 0, + 0x11177, 0x1117F, 0, 0x111E0, 0x111E0, 0, 0x111F5, 0x111FF, 0, 0x11212, 0x11212, 0, + 0x11242, 0x1127F, 0, 0x11287, 0x11287, 0, 0x11289, 0x11289, 0, 0x1128E, 0x1128E, 0, + 0x1129E, 0x1129E, 0, 0x112AA, 0x112AF, 0, 0x112EB, 0x112EF, 0, 0x112FA, 0x112FF, 0, + 0x11304, 0x11304, 0, 0x1130D, 0x1130E, 0, 0x11311, 0x11312, 0, 0x11329, 0x11329, 0, + 0x11331, 0x11331, 0, 0x11334, 0x11334, 0, 0x1133A, 0x1133A, 0, 0x11345, 0x11346, 0, + 0x11349, 0x1134A, 0, 0x1134E, 0x1134F, 0, 0x11351, 0x11356, 0, 0x11358, 0x1135C, 0, + 0x11364, 0x11365, 0, 0x1136D, 0x1136F, 0, 0x11375, 0x1137F, 0, 0x1138A, 0x1138A, 0, + 0x1138C, 0x1138D, 0, 0x1138F, 0x1138F, 0, 0x113B6, 0x113B6, 0, 0x113C1, 0x113C1, 0, + 0x113C3, 0x113C4, 0, 0x113C6, 0x113C6, 0, 0x113CB, 0x113CB, 0, 0x113D6, 0x113D6, 0, + 0x113D9, 0x113E0, 0, 0x113E3, 0x113FF, 0, 0x1145C, 0x1145C, 0, 0x11462, 0x1147F, 0, + 0x114C8, 0x114CF, 0, 0x114DA, 0x1157F, 0, 0x115B6, 0x115B7, 0, 0x115DE, 0x115FF, 0, + 0x11645, 0x1164F, 0, 0x1165A, 0x1165F, 0, 0x1166D, 0x1167F, 0, 0x116BA, 0x116BF, 0, + 0x116CA, 0x116CF, 0, 0x116E4, 0x116FF, 0, 0x1171B, 0x1171C, 0, 0x1172C, 0x1172F, 0, + 0x11747, 0x117FF, 0, 0x1183C, 0x1189F, 0, 0x118F3, 0x118FE, 0, 0x11907, 0x11908, 0, + 0x1190A, 0x1190B, 0, 0x11914, 0x11914, 0, 0x11917, 0x11917, 0, 0x11936, 0x11936, 0, + 0x11939, 0x1193A, 0, 0x11947, 0x1194F, 0, 0x1195A, 0x1199F, 0, 0x119A8, 0x119A9, 0, + 0x119D8, 0x119D9, 0, 0x119E5, 0x119FF, 0, 0x11A48, 0x11A4F, 0, 0x11AA3, 0x11AAF, 0, + 0x11AF9, 0x11AFF, 0, 0x11B0A, 0x11B5F, 0, 0x11B68, 0x11BBF, 0, 0x11BE2, 0x11BEF, 0, + 0x11BFA, 0x11BFF, 0, 0x11C09, 0x11C09, 0, 0x11C37, 0x11C37, 0, 0x11C46, 0x11C4F, 0, + 0x11C6D, 0x11C6F, 0, 0x11C90, 0x11C91, 0, 0x11CA8, 0x11CA8, 0, 0x11CB7, 0x11CFF, 0, + 0x11D07, 0x11D07, 0, 0x11D0A, 0x11D0A, 0, 0x11D37, 0x11D39, 0, 0x11D3B, 0x11D3B, 0, + 0x11D3E, 0x11D3E, 0, 0x11D48, 0x11D4F, 0, 0x11D5A, 0x11D5F, 0, 0x11D66, 0x11D66, 0, + 0x11D69, 0x11D69, 0, 0x11D8F, 0x11D8F, 0, 0x11D92, 0x11D92, 0, 0x11D99, 0x11D9F, 0, + 0x11DAA, 0x11DAF, 0, 0x11DDC, 0x11DDF, 0, 0x11DEA, 0x11EDF, 0, 0x11EF9, 0x11EFF, 0, + 0x11F11, 0x11F11, 0, 0x11F3B, 0x11F3D, 0, 0x11F5B, 0x11FAF, 0, 0x11FB1, 0x11FBF, 0, + 0x11FF2, 0x11FFE, 0, 0x1239A, 0x123FF, 0, 0x1246F, 0x1246F, 0, 0x12475, 0x1247F, 0, + 0x12544, 0x12F8F, 0, 0x12FF3, 0x12FFF, 0, 0x13456, 0x1345F, 0, 0x143FB, 0x143FF, 0, + 0x14647, 0x160FF, 0, 0x1613A, 0x167FF, 0, 0x16A39, 0x16A3F, 0, 0x16A5F, 0x16A5F, 0, + 0x16A6A, 0x16A6D, 0, 0x16ABF, 0x16ABF, 0, 0x16ACA, 0x16ACF, 0, 0x16AEE, 0x16AEF, 0, + 0x16AF6, 0x16AFF, 0, 0x16B46, 0x16B4F, 0, 0x16B5A, 0x16B5A, 0, 0x16B62, 0x16B62, 0, + 0x16B78, 0x16B7C, 0, 0x16B90, 0x16D3F, 0, 0x16D7A, 0x16E3F, 0, 0x16E9B, 0x16E9F, 0, + 0x16EB9, 0x16EBA, 0, 0x16ED4, 0x16EFF, 0, 0x16F4B, 0x16F4E, 0, 0x16F88, 0x16F8E, 0, + 0x16FA0, 0x16FDF, 0, 0x16FE5, 0x16FEF, 0, 0x16FF7, 0x16FFF, 0, 0x18CD6, 0x18CFE, 0, + 0x18D1F, 0x18D7F, 0, 0x18DF3, 0x1AFEF, 0, 0x1AFF4, 0x1AFF4, 0, 0x1AFFC, 0x1AFFC, 0, + 0x1AFFF, 0x1AFFF, 0, 0x1B123, 0x1B131, 0, 0x1B133, 0x1B14F, 0, 0x1B153, 0x1B154, 0, + 0x1B156, 0x1B163, 0, 0x1B168, 0x1B16F, 0, 0x1B2FC, 0x1BBFF, 0, 0x1BC6B, 0x1BC6F, 0, + 0x1BC7D, 0x1BC7F, 0, 0x1BC89, 0x1BC8F, 0, 0x1BC9A, 0x1BC9B, 0, 0x1BCA4, 0x1CBFF, 0, + 0x1CCFD, 0x1CCFF, 0, 0x1CEB4, 0x1CEB9, 0, 0x1CED1, 0x1CEDF, 0, 0x1CEF1, 0x1CEFF, 0, + 0x1CF2E, 0x1CF2F, 0, 0x1CF47, 0x1CF4F, 0, 0x1CFC4, 0x1CFFF, 0, 0x1D0F6, 0x1D0FF, 0, + 0x1D127, 0x1D128, 0, 0x1D1EB, 0x1D1FF, 0, 0x1D246, 0x1D2BF, 0, 0x1D2D4, 0x1D2DF, 0, + 0x1D2F4, 0x1D2FF, 0, 0x1D357, 0x1D35F, 0, 0x1D379, 0x1D3FF, 0, 0x1D455, 0x1D455, 0, + 0x1D49D, 0x1D49D, 0, 0x1D4A0, 0x1D4A1, 0, 0x1D4A3, 0x1D4A4, 0, 0x1D4A7, 0x1D4A8, 0, + 0x1D4AD, 0x1D4AD, 0, 0x1D4BA, 0x1D4BA, 0, 0x1D4BC, 0x1D4BC, 0, 0x1D4C4, 0x1D4C4, 0, + 0x1D506, 0x1D506, 0, 0x1D50B, 0x1D50C, 0, 0x1D515, 0x1D515, 0, 0x1D51D, 0x1D51D, 0, + 0x1D53A, 0x1D53A, 0, 0x1D53F, 0x1D53F, 0, 0x1D545, 0x1D545, 0, 0x1D547, 0x1D549, 0, + 0x1D551, 0x1D551, 0, 0x1D6A6, 0x1D6A7, 0, 0x1D7CC, 0x1D7CD, 0, 0x1DA8C, 0x1DA9A, 0, + 0x1DAA0, 0x1DAA0, 0, 0x1DAB0, 0x1DEFF, 0, 0x1DF1F, 0x1DF24, 0, 0x1DF2B, 0x1DFFF, 0, + 0x1E007, 0x1E007, 0, 0x1E019, 0x1E01A, 0, 0x1E022, 0x1E022, 0, 0x1E025, 0x1E025, 0, + 0x1E02B, 0x1E02F, 0, 0x1E06E, 0x1E08E, 0, 0x1E090, 0x1E0FF, 0, 0x1E12D, 0x1E12F, 0, + 0x1E13E, 0x1E13F, 0, 0x1E14A, 0x1E14D, 0, 0x1E150, 0x1E28F, 0, 0x1E2AF, 0x1E2BF, 0, + 0x1E2FA, 0x1E2FE, 0, 0x1E300, 0x1E4CF, 0, 0x1E4FA, 0x1E5CF, 0, 0x1E5FB, 0x1E5FE, 0, + 0x1E600, 0x1E6BF, 0, 0x1E6DF, 0x1E6DF, 0, 0x1E6F6, 0x1E6FD, 0, 0x1E700, 0x1E7DF, 0, + 0x1E7E7, 0x1E7E7, 0, 0x1E7EC, 0x1E7EC, 0, 0x1E7EF, 0x1E7EF, 0, 0x1E7FF, 0x1E7FF, 0, + 0x1E8C5, 0x1E8C6, 0, 0x1E8D7, 0x1E8FF, 0, 0x1E94C, 0x1E94F, 0, 0x1E95A, 0x1E95D, 0, + 0x1E960, 0x1EC70, 0, 0x1ECB5, 0x1ED00, 0, 0x1ED3E, 0x1EDFF, 0, 0x1EE04, 0x1EE04, 0, + 0x1EE20, 0x1EE20, 0, 0x1EE23, 0x1EE23, 0, 0x1EE25, 0x1EE26, 0, 0x1EE28, 0x1EE28, 0, + 0x1EE33, 0x1EE33, 0, 0x1EE38, 0x1EE38, 0, 0x1EE3A, 0x1EE3A, 0, 0x1EE3C, 0x1EE41, 0, + 0x1EE43, 0x1EE46, 0, 0x1EE48, 0x1EE48, 0, 0x1EE4A, 0x1EE4A, 0, 0x1EE4C, 0x1EE4C, 0, + 0x1EE50, 0x1EE50, 0, 0x1EE53, 0x1EE53, 0, 0x1EE55, 0x1EE56, 0, 0x1EE58, 0x1EE58, 0, + 0x1EE5A, 0x1EE5A, 0, 0x1EE5C, 0x1EE5C, 0, 0x1EE5E, 0x1EE5E, 0, 0x1EE60, 0x1EE60, 0, + 0x1EE63, 0x1EE63, 0, 0x1EE65, 0x1EE66, 0, 0x1EE6B, 0x1EE6B, 0, 0x1EE73, 0x1EE73, 0, + 0x1EE78, 0x1EE78, 0, 0x1EE7D, 0x1EE7D, 0, 0x1EE7F, 0x1EE7F, 0, 0x1EE8A, 0x1EE8A, 0, + 0x1EE9C, 0x1EEA0, 0, 0x1EEA4, 0x1EEA4, 0, 0x1EEAA, 0x1EEAA, 0, 0x1EEBC, 0x1EEEF, 0, + 0x1EEF2, 0x1EFFF, 0, 0x1F02C, 0x1F02F, 0, 0x1F094, 0x1F09F, 0, 0x1F0AF, 0x1F0B0, 0, + 0x1F0C0, 0x1F0C0, 0, 0x1F0D0, 0x1F0D0, 0, 0x1F0F6, 0x1F0FF, 0, 0x1F1AE, 0x1F1E5, 0, + 0x1F203, 0x1F20F, 0, 0x1F23C, 0x1F23F, 0, 0x1F249, 0x1F24F, 0, 0x1F252, 0x1F25F, 0, + 0x1F266, 0x1F2FF, 0, 0x1F6D9, 0x1F6DB, 0, 0x1F6ED, 0x1F6EF, 0, 0x1F6FD, 0x1F6FF, 0, + 0x1F7DA, 0x1F7DF, 0, 0x1F7EC, 0x1F7EF, 0, 0x1F7F1, 0x1F7FF, 0, 0x1F80C, 0x1F80F, 0, + 0x1F848, 0x1F84F, 0, 0x1F85A, 0x1F85F, 0, 0x1F888, 0x1F88F, 0, 0x1F8AE, 0x1F8AF, 0, + 0x1F8BC, 0x1F8BF, 0, 0x1F8C2, 0x1F8CF, 0, 0x1F8D9, 0x1F8FF, 0, 0x1FA58, 0x1FA5F, 0, + 0x1FA6E, 0x1FA6F, 0, 0x1FA7D, 0x1FA7F, 0, 0x1FA8B, 0x1FA8D, 0, 0x1FAC7, 0x1FAC7, 0, + 0x1FAC9, 0x1FACC, 0, 0x1FADD, 0x1FADE, 0, 0x1FAEB, 0x1FAEE, 0, 0x1FAF9, 0x1FAFF, 0, + 0x1FB93, 0x1FB93, 0, 0x1FBFB, 0x1FFFF, 0, 0x2A6E0, 0x2A6FF, 0, 0x2B81E, 0x2B81F, 0, + 0x2CEAE, 0x2CEAF, 0, 0x2EBE1, 0x2EBEF, 0, 0x2EE5E, 0x2F7FF, 0, 0x2FA1E, 0x2FFFF, 0, + 0x3134B, 0x3134F, 0, 0x3347A, 0xE0000, 0, 0xE0002, 0xE001F, 0, 0xE0080, 0xE00FF, 0, + 0xE01F0, 0xEFFFF, 0, 0xFFFFE, 0xFFFFF, 0, 0x10FFFE, 0x10FFFF, 0, 0x41, 0x5A, 1, + 0xC0, 0xD6, 1, 0xD8, 0xDE, 1, 0x100, 0x100, 1, 0x102, 0x102, 1, + 0x104, 0x104, 1, 0x106, 0x106, 1, 0x108, 0x108, 1, 0x10A, 0x10A, 1, + 0x10C, 0x10C, 1, 0x10E, 0x10E, 1, 0x110, 0x110, 1, 0x112, 0x112, 1, + 0x114, 0x114, 1, 0x116, 0x116, 1, 0x118, 0x118, 1, 0x11A, 0x11A, 1, + 0x11C, 0x11C, 1, 0x11E, 0x11E, 1, 0x120, 0x120, 1, 0x122, 0x122, 1, + 0x124, 0x124, 1, 0x126, 0x126, 1, 0x128, 0x128, 1, 0x12A, 0x12A, 1, + 0x12C, 0x12C, 1, 0x12E, 0x12E, 1, 0x130, 0x130, 1, 0x132, 0x132, 1, + 0x134, 0x134, 1, 0x136, 0x136, 1, 0x139, 0x139, 1, 0x13B, 0x13B, 1, + 0x13D, 0x13D, 1, 0x13F, 0x13F, 1, 0x141, 0x141, 1, 0x143, 0x143, 1, + 0x145, 0x145, 1, 0x147, 0x147, 1, 0x14A, 0x14A, 1, 0x14C, 0x14C, 1, + 0x14E, 0x14E, 1, 0x150, 0x150, 1, 0x152, 0x152, 1, 0x154, 0x154, 1, + 0x156, 0x156, 1, 0x158, 0x158, 1, 0x15A, 0x15A, 1, 0x15C, 0x15C, 1, + 0x15E, 0x15E, 1, 0x160, 0x160, 1, 0x162, 0x162, 1, 0x164, 0x164, 1, + 0x166, 0x166, 1, 0x168, 0x168, 1, 0x16A, 0x16A, 1, 0x16C, 0x16C, 1, + 0x16E, 0x16E, 1, 0x170, 0x170, 1, 0x172, 0x172, 1, 0x174, 0x174, 1, + 0x176, 0x176, 1, 0x178, 0x179, 1, 0x17B, 0x17B, 1, 0x17D, 0x17D, 1, + }; + } + + private static int[] rangeChunk2() { + return new int[] { + 0x181, 0x182, 1, 0x184, 0x184, 1, 0x186, 0x187, 1, 0x189, 0x18B, 1, + 0x18E, 0x191, 1, 0x193, 0x194, 1, 0x196, 0x198, 1, 0x19C, 0x19D, 1, + 0x19F, 0x1A0, 1, 0x1A2, 0x1A2, 1, 0x1A4, 0x1A4, 1, 0x1A6, 0x1A7, 1, + 0x1A9, 0x1A9, 1, 0x1AC, 0x1AC, 1, 0x1AE, 0x1AF, 1, 0x1B1, 0x1B3, 1, + 0x1B5, 0x1B5, 1, 0x1B7, 0x1B8, 1, 0x1BC, 0x1BC, 1, 0x1C4, 0x1C4, 1, + 0x1C7, 0x1C7, 1, 0x1CA, 0x1CA, 1, 0x1CD, 0x1CD, 1, 0x1CF, 0x1CF, 1, + 0x1D1, 0x1D1, 1, 0x1D3, 0x1D3, 1, 0x1D5, 0x1D5, 1, 0x1D7, 0x1D7, 1, + 0x1D9, 0x1D9, 1, 0x1DB, 0x1DB, 1, 0x1DE, 0x1DE, 1, 0x1E0, 0x1E0, 1, + 0x1E2, 0x1E2, 1, 0x1E4, 0x1E4, 1, 0x1E6, 0x1E6, 1, 0x1E8, 0x1E8, 1, + 0x1EA, 0x1EA, 1, 0x1EC, 0x1EC, 1, 0x1EE, 0x1EE, 1, 0x1F1, 0x1F1, 1, + 0x1F4, 0x1F4, 1, 0x1F6, 0x1F8, 1, 0x1FA, 0x1FA, 1, 0x1FC, 0x1FC, 1, + 0x1FE, 0x1FE, 1, 0x200, 0x200, 1, 0x202, 0x202, 1, 0x204, 0x204, 1, + 0x206, 0x206, 1, 0x208, 0x208, 1, 0x20A, 0x20A, 1, 0x20C, 0x20C, 1, + 0x20E, 0x20E, 1, 0x210, 0x210, 1, 0x212, 0x212, 1, 0x214, 0x214, 1, + 0x216, 0x216, 1, 0x218, 0x218, 1, 0x21A, 0x21A, 1, 0x21C, 0x21C, 1, + 0x21E, 0x21E, 1, 0x220, 0x220, 1, 0x222, 0x222, 1, 0x224, 0x224, 1, + 0x226, 0x226, 1, 0x228, 0x228, 1, 0x22A, 0x22A, 1, 0x22C, 0x22C, 1, + 0x22E, 0x22E, 1, 0x230, 0x230, 1, 0x232, 0x232, 1, 0x23A, 0x23B, 1, + 0x23D, 0x23E, 1, 0x241, 0x241, 1, 0x243, 0x246, 1, 0x248, 0x248, 1, + 0x24A, 0x24A, 1, 0x24C, 0x24C, 1, 0x24E, 0x24E, 1, 0x370, 0x370, 1, + 0x372, 0x372, 1, 0x376, 0x376, 1, 0x37F, 0x37F, 1, 0x386, 0x386, 1, + 0x388, 0x38A, 1, 0x38C, 0x38C, 1, 0x38E, 0x38F, 1, 0x391, 0x3A1, 1, + 0x3A3, 0x3AB, 1, 0x3CF, 0x3CF, 1, 0x3D2, 0x3D4, 1, 0x3D8, 0x3D8, 1, + 0x3DA, 0x3DA, 1, 0x3DC, 0x3DC, 1, 0x3DE, 0x3DE, 1, 0x3E0, 0x3E0, 1, + 0x3E2, 0x3E2, 1, 0x3E4, 0x3E4, 1, 0x3E6, 0x3E6, 1, 0x3E8, 0x3E8, 1, + 0x3EA, 0x3EA, 1, 0x3EC, 0x3EC, 1, 0x3EE, 0x3EE, 1, 0x3F4, 0x3F4, 1, + 0x3F7, 0x3F7, 1, 0x3F9, 0x3FA, 1, 0x3FD, 0x42F, 1, 0x460, 0x460, 1, + 0x462, 0x462, 1, 0x464, 0x464, 1, 0x466, 0x466, 1, 0x468, 0x468, 1, + 0x46A, 0x46A, 1, 0x46C, 0x46C, 1, 0x46E, 0x46E, 1, 0x470, 0x470, 1, + 0x472, 0x472, 1, 0x474, 0x474, 1, 0x476, 0x476, 1, 0x478, 0x478, 1, + 0x47A, 0x47A, 1, 0x47C, 0x47C, 1, 0x47E, 0x47E, 1, 0x480, 0x480, 1, + 0x48A, 0x48A, 1, 0x48C, 0x48C, 1, 0x48E, 0x48E, 1, 0x490, 0x490, 1, + 0x492, 0x492, 1, 0x494, 0x494, 1, 0x496, 0x496, 1, 0x498, 0x498, 1, + 0x49A, 0x49A, 1, 0x49C, 0x49C, 1, 0x49E, 0x49E, 1, 0x4A0, 0x4A0, 1, + 0x4A2, 0x4A2, 1, 0x4A4, 0x4A4, 1, 0x4A6, 0x4A6, 1, 0x4A8, 0x4A8, 1, + 0x4AA, 0x4AA, 1, 0x4AC, 0x4AC, 1, 0x4AE, 0x4AE, 1, 0x4B0, 0x4B0, 1, + 0x4B2, 0x4B2, 1, 0x4B4, 0x4B4, 1, 0x4B6, 0x4B6, 1, 0x4B8, 0x4B8, 1, + 0x4BA, 0x4BA, 1, 0x4BC, 0x4BC, 1, 0x4BE, 0x4BE, 1, 0x4C0, 0x4C1, 1, + 0x4C3, 0x4C3, 1, 0x4C5, 0x4C5, 1, 0x4C7, 0x4C7, 1, 0x4C9, 0x4C9, 1, + 0x4CB, 0x4CB, 1, 0x4CD, 0x4CD, 1, 0x4D0, 0x4D0, 1, 0x4D2, 0x4D2, 1, + 0x4D4, 0x4D4, 1, 0x4D6, 0x4D6, 1, 0x4D8, 0x4D8, 1, 0x4DA, 0x4DA, 1, + 0x4DC, 0x4DC, 1, 0x4DE, 0x4DE, 1, 0x4E0, 0x4E0, 1, 0x4E2, 0x4E2, 1, + 0x4E4, 0x4E4, 1, 0x4E6, 0x4E6, 1, 0x4E8, 0x4E8, 1, 0x4EA, 0x4EA, 1, + 0x4EC, 0x4EC, 1, 0x4EE, 0x4EE, 1, 0x4F0, 0x4F0, 1, 0x4F2, 0x4F2, 1, + 0x4F4, 0x4F4, 1, 0x4F6, 0x4F6, 1, 0x4F8, 0x4F8, 1, 0x4FA, 0x4FA, 1, + 0x4FC, 0x4FC, 1, 0x4FE, 0x4FE, 1, 0x500, 0x500, 1, 0x502, 0x502, 1, + 0x504, 0x504, 1, 0x506, 0x506, 1, 0x508, 0x508, 1, 0x50A, 0x50A, 1, + 0x50C, 0x50C, 1, 0x50E, 0x50E, 1, 0x510, 0x510, 1, 0x512, 0x512, 1, + 0x514, 0x514, 1, 0x516, 0x516, 1, 0x518, 0x518, 1, 0x51A, 0x51A, 1, + 0x51C, 0x51C, 1, 0x51E, 0x51E, 1, 0x520, 0x520, 1, 0x522, 0x522, 1, + 0x524, 0x524, 1, 0x526, 0x526, 1, 0x528, 0x528, 1, 0x52A, 0x52A, 1, + 0x52C, 0x52C, 1, 0x52E, 0x52E, 1, 0x531, 0x556, 1, 0x10A0, 0x10C5, 1, + 0x10C7, 0x10C7, 1, 0x10CD, 0x10CD, 1, 0x13A0, 0x13F5, 1, 0x1C89, 0x1C89, 1, + 0x1C90, 0x1CBA, 1, 0x1CBD, 0x1CBF, 1, 0x1E00, 0x1E00, 1, 0x1E02, 0x1E02, 1, + 0x1E04, 0x1E04, 1, 0x1E06, 0x1E06, 1, 0x1E08, 0x1E08, 1, 0x1E0A, 0x1E0A, 1, + 0x1E0C, 0x1E0C, 1, 0x1E0E, 0x1E0E, 1, 0x1E10, 0x1E10, 1, 0x1E12, 0x1E12, 1, + 0x1E14, 0x1E14, 1, 0x1E16, 0x1E16, 1, 0x1E18, 0x1E18, 1, 0x1E1A, 0x1E1A, 1, + 0x1E1C, 0x1E1C, 1, 0x1E1E, 0x1E1E, 1, 0x1E20, 0x1E20, 1, 0x1E22, 0x1E22, 1, + 0x1E24, 0x1E24, 1, 0x1E26, 0x1E26, 1, 0x1E28, 0x1E28, 1, 0x1E2A, 0x1E2A, 1, + 0x1E2C, 0x1E2C, 1, 0x1E2E, 0x1E2E, 1, 0x1E30, 0x1E30, 1, 0x1E32, 0x1E32, 1, + 0x1E34, 0x1E34, 1, 0x1E36, 0x1E36, 1, 0x1E38, 0x1E38, 1, 0x1E3A, 0x1E3A, 1, + 0x1E3C, 0x1E3C, 1, 0x1E3E, 0x1E3E, 1, 0x1E40, 0x1E40, 1, 0x1E42, 0x1E42, 1, + 0x1E44, 0x1E44, 1, 0x1E46, 0x1E46, 1, 0x1E48, 0x1E48, 1, 0x1E4A, 0x1E4A, 1, + 0x1E4C, 0x1E4C, 1, 0x1E4E, 0x1E4E, 1, 0x1E50, 0x1E50, 1, 0x1E52, 0x1E52, 1, + 0x1E54, 0x1E54, 1, 0x1E56, 0x1E56, 1, 0x1E58, 0x1E58, 1, 0x1E5A, 0x1E5A, 1, + 0x1E5C, 0x1E5C, 1, 0x1E5E, 0x1E5E, 1, 0x1E60, 0x1E60, 1, 0x1E62, 0x1E62, 1, + 0x1E64, 0x1E64, 1, 0x1E66, 0x1E66, 1, 0x1E68, 0x1E68, 1, 0x1E6A, 0x1E6A, 1, + 0x1E6C, 0x1E6C, 1, 0x1E6E, 0x1E6E, 1, 0x1E70, 0x1E70, 1, 0x1E72, 0x1E72, 1, + 0x1E74, 0x1E74, 1, 0x1E76, 0x1E76, 1, 0x1E78, 0x1E78, 1, 0x1E7A, 0x1E7A, 1, + 0x1E7C, 0x1E7C, 1, 0x1E7E, 0x1E7E, 1, 0x1E80, 0x1E80, 1, 0x1E82, 0x1E82, 1, + 0x1E84, 0x1E84, 1, 0x1E86, 0x1E86, 1, 0x1E88, 0x1E88, 1, 0x1E8A, 0x1E8A, 1, + 0x1E8C, 0x1E8C, 1, 0x1E8E, 0x1E8E, 1, 0x1E90, 0x1E90, 1, 0x1E92, 0x1E92, 1, + 0x1E94, 0x1E94, 1, 0x1E9E, 0x1E9E, 1, 0x1EA0, 0x1EA0, 1, 0x1EA2, 0x1EA2, 1, + 0x1EA4, 0x1EA4, 1, 0x1EA6, 0x1EA6, 1, 0x1EA8, 0x1EA8, 1, 0x1EAA, 0x1EAA, 1, + 0x1EAC, 0x1EAC, 1, 0x1EAE, 0x1EAE, 1, 0x1EB0, 0x1EB0, 1, 0x1EB2, 0x1EB2, 1, + 0x1EB4, 0x1EB4, 1, 0x1EB6, 0x1EB6, 1, 0x1EB8, 0x1EB8, 1, 0x1EBA, 0x1EBA, 1, + 0x1EBC, 0x1EBC, 1, 0x1EBE, 0x1EBE, 1, 0x1EC0, 0x1EC0, 1, 0x1EC2, 0x1EC2, 1, + 0x1EC4, 0x1EC4, 1, 0x1EC6, 0x1EC6, 1, 0x1EC8, 0x1EC8, 1, 0x1ECA, 0x1ECA, 1, + 0x1ECC, 0x1ECC, 1, 0x1ECE, 0x1ECE, 1, 0x1ED0, 0x1ED0, 1, 0x1ED2, 0x1ED2, 1, + 0x1ED4, 0x1ED4, 1, 0x1ED6, 0x1ED6, 1, 0x1ED8, 0x1ED8, 1, 0x1EDA, 0x1EDA, 1, + 0x1EDC, 0x1EDC, 1, 0x1EDE, 0x1EDE, 1, 0x1EE0, 0x1EE0, 1, 0x1EE2, 0x1EE2, 1, + 0x1EE4, 0x1EE4, 1, 0x1EE6, 0x1EE6, 1, 0x1EE8, 0x1EE8, 1, 0x1EEA, 0x1EEA, 1, + 0x1EEC, 0x1EEC, 1, 0x1EEE, 0x1EEE, 1, 0x1EF0, 0x1EF0, 1, 0x1EF2, 0x1EF2, 1, + 0x1EF4, 0x1EF4, 1, 0x1EF6, 0x1EF6, 1, 0x1EF8, 0x1EF8, 1, 0x1EFA, 0x1EFA, 1, + 0x1EFC, 0x1EFC, 1, 0x1EFE, 0x1EFE, 1, 0x1F08, 0x1F0F, 1, 0x1F18, 0x1F1D, 1, + 0x1F28, 0x1F2F, 1, 0x1F38, 0x1F3F, 1, 0x1F48, 0x1F4D, 1, 0x1F59, 0x1F59, 1, + 0x1F5B, 0x1F5B, 1, 0x1F5D, 0x1F5D, 1, 0x1F5F, 0x1F5F, 1, 0x1F68, 0x1F6F, 1, + 0x1FB8, 0x1FBB, 1, 0x1FC8, 0x1FCB, 1, 0x1FD8, 0x1FDB, 1, 0x1FE8, 0x1FEC, 1, + 0x1FF8, 0x1FFB, 1, 0x2102, 0x2102, 1, 0x2107, 0x2107, 1, 0x210B, 0x210D, 1, + 0x2110, 0x2112, 1, 0x2115, 0x2115, 1, 0x2119, 0x211D, 1, 0x2124, 0x2124, 1, + 0x2126, 0x2126, 1, 0x2128, 0x2128, 1, 0x212A, 0x212D, 1, 0x2130, 0x2133, 1, + 0x213E, 0x213F, 1, 0x2145, 0x2145, 1, 0x2183, 0x2183, 1, 0x2C00, 0x2C2F, 1, + 0x2C60, 0x2C60, 1, 0x2C62, 0x2C64, 1, 0x2C67, 0x2C67, 1, 0x2C69, 0x2C69, 1, + 0x2C6B, 0x2C6B, 1, 0x2C6D, 0x2C70, 1, 0x2C72, 0x2C72, 1, 0x2C75, 0x2C75, 1, + 0x2C7E, 0x2C80, 1, 0x2C82, 0x2C82, 1, 0x2C84, 0x2C84, 1, 0x2C86, 0x2C86, 1, + 0x2C88, 0x2C88, 1, 0x2C8A, 0x2C8A, 1, 0x2C8C, 0x2C8C, 1, 0x2C8E, 0x2C8E, 1, + 0x2C90, 0x2C90, 1, 0x2C92, 0x2C92, 1, 0x2C94, 0x2C94, 1, 0x2C96, 0x2C96, 1, + 0x2C98, 0x2C98, 1, 0x2C9A, 0x2C9A, 1, 0x2C9C, 0x2C9C, 1, 0x2C9E, 0x2C9E, 1, + 0x2CA0, 0x2CA0, 1, 0x2CA2, 0x2CA2, 1, 0x2CA4, 0x2CA4, 1, 0x2CA6, 0x2CA6, 1, + 0x2CA8, 0x2CA8, 1, 0x2CAA, 0x2CAA, 1, 0x2CAC, 0x2CAC, 1, 0x2CAE, 0x2CAE, 1, + }; + } + + private static int[] rangeChunk3() { + return new int[] { + 0x2CB0, 0x2CB0, 1, 0x2CB2, 0x2CB2, 1, 0x2CB4, 0x2CB4, 1, 0x2CB6, 0x2CB6, 1, + 0x2CB8, 0x2CB8, 1, 0x2CBA, 0x2CBA, 1, 0x2CBC, 0x2CBC, 1, 0x2CBE, 0x2CBE, 1, + 0x2CC0, 0x2CC0, 1, 0x2CC2, 0x2CC2, 1, 0x2CC4, 0x2CC4, 1, 0x2CC6, 0x2CC6, 1, + 0x2CC8, 0x2CC8, 1, 0x2CCA, 0x2CCA, 1, 0x2CCC, 0x2CCC, 1, 0x2CCE, 0x2CCE, 1, + 0x2CD0, 0x2CD0, 1, 0x2CD2, 0x2CD2, 1, 0x2CD4, 0x2CD4, 1, 0x2CD6, 0x2CD6, 1, + 0x2CD8, 0x2CD8, 1, 0x2CDA, 0x2CDA, 1, 0x2CDC, 0x2CDC, 1, 0x2CDE, 0x2CDE, 1, + 0x2CE0, 0x2CE0, 1, 0x2CE2, 0x2CE2, 1, 0x2CEB, 0x2CEB, 1, 0x2CED, 0x2CED, 1, + 0x2CF2, 0x2CF2, 1, 0xA640, 0xA640, 1, 0xA642, 0xA642, 1, 0xA644, 0xA644, 1, + 0xA646, 0xA646, 1, 0xA648, 0xA648, 1, 0xA64A, 0xA64A, 1, 0xA64C, 0xA64C, 1, + 0xA64E, 0xA64E, 1, 0xA650, 0xA650, 1, 0xA652, 0xA652, 1, 0xA654, 0xA654, 1, + 0xA656, 0xA656, 1, 0xA658, 0xA658, 1, 0xA65A, 0xA65A, 1, 0xA65C, 0xA65C, 1, + 0xA65E, 0xA65E, 1, 0xA660, 0xA660, 1, 0xA662, 0xA662, 1, 0xA664, 0xA664, 1, + 0xA666, 0xA666, 1, 0xA668, 0xA668, 1, 0xA66A, 0xA66A, 1, 0xA66C, 0xA66C, 1, + 0xA680, 0xA680, 1, 0xA682, 0xA682, 1, 0xA684, 0xA684, 1, 0xA686, 0xA686, 1, + 0xA688, 0xA688, 1, 0xA68A, 0xA68A, 1, 0xA68C, 0xA68C, 1, 0xA68E, 0xA68E, 1, + 0xA690, 0xA690, 1, 0xA692, 0xA692, 1, 0xA694, 0xA694, 1, 0xA696, 0xA696, 1, + 0xA698, 0xA698, 1, 0xA69A, 0xA69A, 1, 0xA722, 0xA722, 1, 0xA724, 0xA724, 1, + 0xA726, 0xA726, 1, 0xA728, 0xA728, 1, 0xA72A, 0xA72A, 1, 0xA72C, 0xA72C, 1, + 0xA72E, 0xA72E, 1, 0xA732, 0xA732, 1, 0xA734, 0xA734, 1, 0xA736, 0xA736, 1, + 0xA738, 0xA738, 1, 0xA73A, 0xA73A, 1, 0xA73C, 0xA73C, 1, 0xA73E, 0xA73E, 1, + 0xA740, 0xA740, 1, 0xA742, 0xA742, 1, 0xA744, 0xA744, 1, 0xA746, 0xA746, 1, + 0xA748, 0xA748, 1, 0xA74A, 0xA74A, 1, 0xA74C, 0xA74C, 1, 0xA74E, 0xA74E, 1, + 0xA750, 0xA750, 1, 0xA752, 0xA752, 1, 0xA754, 0xA754, 1, 0xA756, 0xA756, 1, + 0xA758, 0xA758, 1, 0xA75A, 0xA75A, 1, 0xA75C, 0xA75C, 1, 0xA75E, 0xA75E, 1, + 0xA760, 0xA760, 1, 0xA762, 0xA762, 1, 0xA764, 0xA764, 1, 0xA766, 0xA766, 1, + 0xA768, 0xA768, 1, 0xA76A, 0xA76A, 1, 0xA76C, 0xA76C, 1, 0xA76E, 0xA76E, 1, + 0xA779, 0xA779, 1, 0xA77B, 0xA77B, 1, 0xA77D, 0xA77E, 1, 0xA780, 0xA780, 1, + 0xA782, 0xA782, 1, 0xA784, 0xA784, 1, 0xA786, 0xA786, 1, 0xA78B, 0xA78B, 1, + 0xA78D, 0xA78D, 1, 0xA790, 0xA790, 1, 0xA792, 0xA792, 1, 0xA796, 0xA796, 1, + 0xA798, 0xA798, 1, 0xA79A, 0xA79A, 1, 0xA79C, 0xA79C, 1, 0xA79E, 0xA79E, 1, + 0xA7A0, 0xA7A0, 1, 0xA7A2, 0xA7A2, 1, 0xA7A4, 0xA7A4, 1, 0xA7A6, 0xA7A6, 1, + 0xA7A8, 0xA7A8, 1, 0xA7AA, 0xA7AE, 1, 0xA7B0, 0xA7B4, 1, 0xA7B6, 0xA7B6, 1, + 0xA7B8, 0xA7B8, 1, 0xA7BA, 0xA7BA, 1, 0xA7BC, 0xA7BC, 1, 0xA7BE, 0xA7BE, 1, + 0xA7C0, 0xA7C0, 1, 0xA7C2, 0xA7C2, 1, 0xA7C4, 0xA7C7, 1, 0xA7C9, 0xA7C9, 1, + 0xA7CB, 0xA7CC, 1, 0xA7CE, 0xA7CE, 1, 0xA7D0, 0xA7D0, 1, 0xA7D2, 0xA7D2, 1, + 0xA7D4, 0xA7D4, 1, 0xA7D6, 0xA7D6, 1, 0xA7D8, 0xA7D8, 1, 0xA7DA, 0xA7DA, 1, + 0xA7DC, 0xA7DC, 1, 0xA7F5, 0xA7F5, 1, 0xFF21, 0xFF3A, 1, 0x10400, 0x10427, 1, + 0x104B0, 0x104D3, 1, 0x10570, 0x1057A, 1, 0x1057C, 0x1058A, 1, 0x1058C, 0x10592, 1, + 0x10594, 0x10595, 1, 0x10C80, 0x10CB2, 1, 0x10D50, 0x10D65, 1, 0x118A0, 0x118BF, 1, + 0x16E40, 0x16E5F, 1, 0x16EA0, 0x16EB8, 1, 0x1D400, 0x1D419, 1, 0x1D434, 0x1D44D, 1, + 0x1D468, 0x1D481, 1, 0x1D49C, 0x1D49C, 1, 0x1D49E, 0x1D49F, 1, 0x1D4A2, 0x1D4A2, 1, + 0x1D4A5, 0x1D4A6, 1, 0x1D4A9, 0x1D4AC, 1, 0x1D4AE, 0x1D4B5, 1, 0x1D4D0, 0x1D4E9, 1, + 0x1D504, 0x1D505, 1, 0x1D507, 0x1D50A, 1, 0x1D50D, 0x1D514, 1, 0x1D516, 0x1D51C, 1, + 0x1D538, 0x1D539, 1, 0x1D53B, 0x1D53E, 1, 0x1D540, 0x1D544, 1, 0x1D546, 0x1D546, 1, + 0x1D54A, 0x1D550, 1, 0x1D56C, 0x1D585, 1, 0x1D5A0, 0x1D5B9, 1, 0x1D5D4, 0x1D5ED, 1, + 0x1D608, 0x1D621, 1, 0x1D63C, 0x1D655, 1, 0x1D670, 0x1D689, 1, 0x1D6A8, 0x1D6C0, 1, + 0x1D6E2, 0x1D6FA, 1, 0x1D71C, 0x1D734, 1, 0x1D756, 0x1D76E, 1, 0x1D790, 0x1D7A8, 1, + 0x1D7CA, 0x1D7CA, 1, 0x1E900, 0x1E921, 1, 0x61, 0x7A, 2, 0xB5, 0xB5, 2, + 0xDF, 0xF6, 2, 0xF8, 0xFF, 2, 0x101, 0x101, 2, 0x103, 0x103, 2, + 0x105, 0x105, 2, 0x107, 0x107, 2, 0x109, 0x109, 2, 0x10B, 0x10B, 2, + 0x10D, 0x10D, 2, 0x10F, 0x10F, 2, 0x111, 0x111, 2, 0x113, 0x113, 2, + 0x115, 0x115, 2, 0x117, 0x117, 2, 0x119, 0x119, 2, 0x11B, 0x11B, 2, + 0x11D, 0x11D, 2, 0x11F, 0x11F, 2, 0x121, 0x121, 2, 0x123, 0x123, 2, + 0x125, 0x125, 2, 0x127, 0x127, 2, 0x129, 0x129, 2, 0x12B, 0x12B, 2, + 0x12D, 0x12D, 2, 0x12F, 0x12F, 2, 0x131, 0x131, 2, 0x133, 0x133, 2, + 0x135, 0x135, 2, 0x137, 0x138, 2, 0x13A, 0x13A, 2, 0x13C, 0x13C, 2, + 0x13E, 0x13E, 2, 0x140, 0x140, 2, 0x142, 0x142, 2, 0x144, 0x144, 2, + 0x146, 0x146, 2, 0x148, 0x149, 2, 0x14B, 0x14B, 2, 0x14D, 0x14D, 2, + 0x14F, 0x14F, 2, 0x151, 0x151, 2, 0x153, 0x153, 2, 0x155, 0x155, 2, + 0x157, 0x157, 2, 0x159, 0x159, 2, 0x15B, 0x15B, 2, 0x15D, 0x15D, 2, + 0x15F, 0x15F, 2, 0x161, 0x161, 2, 0x163, 0x163, 2, 0x165, 0x165, 2, + 0x167, 0x167, 2, 0x169, 0x169, 2, 0x16B, 0x16B, 2, 0x16D, 0x16D, 2, + 0x16F, 0x16F, 2, 0x171, 0x171, 2, 0x173, 0x173, 2, 0x175, 0x175, 2, + 0x177, 0x177, 2, 0x17A, 0x17A, 2, 0x17C, 0x17C, 2, 0x17E, 0x180, 2, + 0x183, 0x183, 2, 0x185, 0x185, 2, 0x188, 0x188, 2, 0x18C, 0x18D, 2, + 0x192, 0x192, 2, 0x195, 0x195, 2, 0x199, 0x19B, 2, 0x19E, 0x19E, 2, + 0x1A1, 0x1A1, 2, 0x1A3, 0x1A3, 2, 0x1A5, 0x1A5, 2, 0x1A8, 0x1A8, 2, + 0x1AA, 0x1AB, 2, 0x1AD, 0x1AD, 2, 0x1B0, 0x1B0, 2, 0x1B4, 0x1B4, 2, + 0x1B6, 0x1B6, 2, 0x1B9, 0x1BA, 2, 0x1BD, 0x1BF, 2, 0x1C6, 0x1C6, 2, + 0x1C9, 0x1C9, 2, 0x1CC, 0x1CC, 2, 0x1CE, 0x1CE, 2, 0x1D0, 0x1D0, 2, + 0x1D2, 0x1D2, 2, 0x1D4, 0x1D4, 2, 0x1D6, 0x1D6, 2, 0x1D8, 0x1D8, 2, + 0x1DA, 0x1DA, 2, 0x1DC, 0x1DD, 2, 0x1DF, 0x1DF, 2, 0x1E1, 0x1E1, 2, + 0x1E3, 0x1E3, 2, 0x1E5, 0x1E5, 2, 0x1E7, 0x1E7, 2, 0x1E9, 0x1E9, 2, + 0x1EB, 0x1EB, 2, 0x1ED, 0x1ED, 2, 0x1EF, 0x1F0, 2, 0x1F3, 0x1F3, 2, + 0x1F5, 0x1F5, 2, 0x1F9, 0x1F9, 2, 0x1FB, 0x1FB, 2, 0x1FD, 0x1FD, 2, + 0x1FF, 0x1FF, 2, 0x201, 0x201, 2, 0x203, 0x203, 2, 0x205, 0x205, 2, + 0x207, 0x207, 2, 0x209, 0x209, 2, 0x20B, 0x20B, 2, 0x20D, 0x20D, 2, + 0x20F, 0x20F, 2, 0x211, 0x211, 2, 0x213, 0x213, 2, 0x215, 0x215, 2, + 0x217, 0x217, 2, 0x219, 0x219, 2, 0x21B, 0x21B, 2, 0x21D, 0x21D, 2, + 0x21F, 0x21F, 2, 0x221, 0x221, 2, 0x223, 0x223, 2, 0x225, 0x225, 2, + 0x227, 0x227, 2, 0x229, 0x229, 2, 0x22B, 0x22B, 2, 0x22D, 0x22D, 2, + 0x22F, 0x22F, 2, 0x231, 0x231, 2, 0x233, 0x239, 2, 0x23C, 0x23C, 2, + 0x23F, 0x240, 2, 0x242, 0x242, 2, 0x247, 0x247, 2, 0x249, 0x249, 2, + 0x24B, 0x24B, 2, 0x24D, 0x24D, 2, 0x24F, 0x293, 2, 0x296, 0x2AF, 2, + 0x371, 0x371, 2, 0x373, 0x373, 2, 0x377, 0x377, 2, 0x37B, 0x37D, 2, + 0x390, 0x390, 2, 0x3AC, 0x3CE, 2, 0x3D0, 0x3D1, 2, 0x3D5, 0x3D7, 2, + 0x3D9, 0x3D9, 2, 0x3DB, 0x3DB, 2, 0x3DD, 0x3DD, 2, 0x3DF, 0x3DF, 2, + 0x3E1, 0x3E1, 2, 0x3E3, 0x3E3, 2, 0x3E5, 0x3E5, 2, 0x3E7, 0x3E7, 2, + 0x3E9, 0x3E9, 2, 0x3EB, 0x3EB, 2, 0x3ED, 0x3ED, 2, 0x3EF, 0x3F3, 2, + 0x3F5, 0x3F5, 2, 0x3F8, 0x3F8, 2, 0x3FB, 0x3FC, 2, 0x430, 0x45F, 2, + 0x461, 0x461, 2, 0x463, 0x463, 2, 0x465, 0x465, 2, 0x467, 0x467, 2, + 0x469, 0x469, 2, 0x46B, 0x46B, 2, 0x46D, 0x46D, 2, 0x46F, 0x46F, 2, + 0x471, 0x471, 2, 0x473, 0x473, 2, 0x475, 0x475, 2, 0x477, 0x477, 2, + 0x479, 0x479, 2, 0x47B, 0x47B, 2, 0x47D, 0x47D, 2, 0x47F, 0x47F, 2, + 0x481, 0x481, 2, 0x48B, 0x48B, 2, 0x48D, 0x48D, 2, 0x48F, 0x48F, 2, + 0x491, 0x491, 2, 0x493, 0x493, 2, 0x495, 0x495, 2, 0x497, 0x497, 2, + 0x499, 0x499, 2, 0x49B, 0x49B, 2, 0x49D, 0x49D, 2, 0x49F, 0x49F, 2, + 0x4A1, 0x4A1, 2, 0x4A3, 0x4A3, 2, 0x4A5, 0x4A5, 2, 0x4A7, 0x4A7, 2, + 0x4A9, 0x4A9, 2, 0x4AB, 0x4AB, 2, 0x4AD, 0x4AD, 2, 0x4AF, 0x4AF, 2, + 0x4B1, 0x4B1, 2, 0x4B3, 0x4B3, 2, 0x4B5, 0x4B5, 2, 0x4B7, 0x4B7, 2, + }; + } + + private static int[] rangeChunk4() { + return new int[] { + 0x4B9, 0x4B9, 2, 0x4BB, 0x4BB, 2, 0x4BD, 0x4BD, 2, 0x4BF, 0x4BF, 2, + 0x4C2, 0x4C2, 2, 0x4C4, 0x4C4, 2, 0x4C6, 0x4C6, 2, 0x4C8, 0x4C8, 2, + 0x4CA, 0x4CA, 2, 0x4CC, 0x4CC, 2, 0x4CE, 0x4CF, 2, 0x4D1, 0x4D1, 2, + 0x4D3, 0x4D3, 2, 0x4D5, 0x4D5, 2, 0x4D7, 0x4D7, 2, 0x4D9, 0x4D9, 2, + 0x4DB, 0x4DB, 2, 0x4DD, 0x4DD, 2, 0x4DF, 0x4DF, 2, 0x4E1, 0x4E1, 2, + 0x4E3, 0x4E3, 2, 0x4E5, 0x4E5, 2, 0x4E7, 0x4E7, 2, 0x4E9, 0x4E9, 2, + 0x4EB, 0x4EB, 2, 0x4ED, 0x4ED, 2, 0x4EF, 0x4EF, 2, 0x4F1, 0x4F1, 2, + 0x4F3, 0x4F3, 2, 0x4F5, 0x4F5, 2, 0x4F7, 0x4F7, 2, 0x4F9, 0x4F9, 2, + 0x4FB, 0x4FB, 2, 0x4FD, 0x4FD, 2, 0x4FF, 0x4FF, 2, 0x501, 0x501, 2, + 0x503, 0x503, 2, 0x505, 0x505, 2, 0x507, 0x507, 2, 0x509, 0x509, 2, + 0x50B, 0x50B, 2, 0x50D, 0x50D, 2, 0x50F, 0x50F, 2, 0x511, 0x511, 2, + 0x513, 0x513, 2, 0x515, 0x515, 2, 0x517, 0x517, 2, 0x519, 0x519, 2, + 0x51B, 0x51B, 2, 0x51D, 0x51D, 2, 0x51F, 0x51F, 2, 0x521, 0x521, 2, + 0x523, 0x523, 2, 0x525, 0x525, 2, 0x527, 0x527, 2, 0x529, 0x529, 2, + 0x52B, 0x52B, 2, 0x52D, 0x52D, 2, 0x52F, 0x52F, 2, 0x560, 0x588, 2, + 0x10D0, 0x10FA, 2, 0x10FD, 0x10FF, 2, 0x13F8, 0x13FD, 2, 0x1C80, 0x1C88, 2, + 0x1C8A, 0x1C8A, 2, 0x1D00, 0x1D2B, 2, 0x1D6B, 0x1D77, 2, 0x1D79, 0x1D9A, 2, + 0x1E01, 0x1E01, 2, 0x1E03, 0x1E03, 2, 0x1E05, 0x1E05, 2, 0x1E07, 0x1E07, 2, + 0x1E09, 0x1E09, 2, 0x1E0B, 0x1E0B, 2, 0x1E0D, 0x1E0D, 2, 0x1E0F, 0x1E0F, 2, + 0x1E11, 0x1E11, 2, 0x1E13, 0x1E13, 2, 0x1E15, 0x1E15, 2, 0x1E17, 0x1E17, 2, + 0x1E19, 0x1E19, 2, 0x1E1B, 0x1E1B, 2, 0x1E1D, 0x1E1D, 2, 0x1E1F, 0x1E1F, 2, + 0x1E21, 0x1E21, 2, 0x1E23, 0x1E23, 2, 0x1E25, 0x1E25, 2, 0x1E27, 0x1E27, 2, + 0x1E29, 0x1E29, 2, 0x1E2B, 0x1E2B, 2, 0x1E2D, 0x1E2D, 2, 0x1E2F, 0x1E2F, 2, + 0x1E31, 0x1E31, 2, 0x1E33, 0x1E33, 2, 0x1E35, 0x1E35, 2, 0x1E37, 0x1E37, 2, + 0x1E39, 0x1E39, 2, 0x1E3B, 0x1E3B, 2, 0x1E3D, 0x1E3D, 2, 0x1E3F, 0x1E3F, 2, + 0x1E41, 0x1E41, 2, 0x1E43, 0x1E43, 2, 0x1E45, 0x1E45, 2, 0x1E47, 0x1E47, 2, + 0x1E49, 0x1E49, 2, 0x1E4B, 0x1E4B, 2, 0x1E4D, 0x1E4D, 2, 0x1E4F, 0x1E4F, 2, + 0x1E51, 0x1E51, 2, 0x1E53, 0x1E53, 2, 0x1E55, 0x1E55, 2, 0x1E57, 0x1E57, 2, + 0x1E59, 0x1E59, 2, 0x1E5B, 0x1E5B, 2, 0x1E5D, 0x1E5D, 2, 0x1E5F, 0x1E5F, 2, + 0x1E61, 0x1E61, 2, 0x1E63, 0x1E63, 2, 0x1E65, 0x1E65, 2, 0x1E67, 0x1E67, 2, + 0x1E69, 0x1E69, 2, 0x1E6B, 0x1E6B, 2, 0x1E6D, 0x1E6D, 2, 0x1E6F, 0x1E6F, 2, + 0x1E71, 0x1E71, 2, 0x1E73, 0x1E73, 2, 0x1E75, 0x1E75, 2, 0x1E77, 0x1E77, 2, + 0x1E79, 0x1E79, 2, 0x1E7B, 0x1E7B, 2, 0x1E7D, 0x1E7D, 2, 0x1E7F, 0x1E7F, 2, + 0x1E81, 0x1E81, 2, 0x1E83, 0x1E83, 2, 0x1E85, 0x1E85, 2, 0x1E87, 0x1E87, 2, + 0x1E89, 0x1E89, 2, 0x1E8B, 0x1E8B, 2, 0x1E8D, 0x1E8D, 2, 0x1E8F, 0x1E8F, 2, + 0x1E91, 0x1E91, 2, 0x1E93, 0x1E93, 2, 0x1E95, 0x1E9D, 2, 0x1E9F, 0x1E9F, 2, + 0x1EA1, 0x1EA1, 2, 0x1EA3, 0x1EA3, 2, 0x1EA5, 0x1EA5, 2, 0x1EA7, 0x1EA7, 2, + 0x1EA9, 0x1EA9, 2, 0x1EAB, 0x1EAB, 2, 0x1EAD, 0x1EAD, 2, 0x1EAF, 0x1EAF, 2, + 0x1EB1, 0x1EB1, 2, 0x1EB3, 0x1EB3, 2, 0x1EB5, 0x1EB5, 2, 0x1EB7, 0x1EB7, 2, + 0x1EB9, 0x1EB9, 2, 0x1EBB, 0x1EBB, 2, 0x1EBD, 0x1EBD, 2, 0x1EBF, 0x1EBF, 2, + 0x1EC1, 0x1EC1, 2, 0x1EC3, 0x1EC3, 2, 0x1EC5, 0x1EC5, 2, 0x1EC7, 0x1EC7, 2, + 0x1EC9, 0x1EC9, 2, 0x1ECB, 0x1ECB, 2, 0x1ECD, 0x1ECD, 2, 0x1ECF, 0x1ECF, 2, + 0x1ED1, 0x1ED1, 2, 0x1ED3, 0x1ED3, 2, 0x1ED5, 0x1ED5, 2, 0x1ED7, 0x1ED7, 2, + 0x1ED9, 0x1ED9, 2, 0x1EDB, 0x1EDB, 2, 0x1EDD, 0x1EDD, 2, 0x1EDF, 0x1EDF, 2, + 0x1EE1, 0x1EE1, 2, 0x1EE3, 0x1EE3, 2, 0x1EE5, 0x1EE5, 2, 0x1EE7, 0x1EE7, 2, + 0x1EE9, 0x1EE9, 2, 0x1EEB, 0x1EEB, 2, 0x1EED, 0x1EED, 2, 0x1EEF, 0x1EEF, 2, + 0x1EF1, 0x1EF1, 2, 0x1EF3, 0x1EF3, 2, 0x1EF5, 0x1EF5, 2, 0x1EF7, 0x1EF7, 2, + 0x1EF9, 0x1EF9, 2, 0x1EFB, 0x1EFB, 2, 0x1EFD, 0x1EFD, 2, 0x1EFF, 0x1F07, 2, + 0x1F10, 0x1F15, 2, 0x1F20, 0x1F27, 2, 0x1F30, 0x1F37, 2, 0x1F40, 0x1F45, 2, + 0x1F50, 0x1F57, 2, 0x1F60, 0x1F67, 2, 0x1F70, 0x1F7D, 2, 0x1F80, 0x1F87, 2, + 0x1F90, 0x1F97, 2, 0x1FA0, 0x1FA7, 2, 0x1FB0, 0x1FB4, 2, 0x1FB6, 0x1FB7, 2, + 0x1FBE, 0x1FBE, 2, 0x1FC2, 0x1FC4, 2, 0x1FC6, 0x1FC7, 2, 0x1FD0, 0x1FD3, 2, + 0x1FD6, 0x1FD7, 2, 0x1FE0, 0x1FE7, 2, 0x1FF2, 0x1FF4, 2, 0x1FF6, 0x1FF7, 2, + 0x210A, 0x210A, 2, 0x210E, 0x210F, 2, 0x2113, 0x2113, 2, 0x212F, 0x212F, 2, + 0x2134, 0x2134, 2, 0x2139, 0x2139, 2, 0x213C, 0x213D, 2, 0x2146, 0x2149, 2, + 0x214E, 0x214E, 2, 0x2184, 0x2184, 2, 0x2C30, 0x2C5F, 2, 0x2C61, 0x2C61, 2, + 0x2C65, 0x2C66, 2, 0x2C68, 0x2C68, 2, 0x2C6A, 0x2C6A, 2, 0x2C6C, 0x2C6C, 2, + 0x2C71, 0x2C71, 2, 0x2C73, 0x2C74, 2, 0x2C76, 0x2C7B, 2, 0x2C81, 0x2C81, 2, + 0x2C83, 0x2C83, 2, 0x2C85, 0x2C85, 2, 0x2C87, 0x2C87, 2, 0x2C89, 0x2C89, 2, + 0x2C8B, 0x2C8B, 2, 0x2C8D, 0x2C8D, 2, 0x2C8F, 0x2C8F, 2, 0x2C91, 0x2C91, 2, + 0x2C93, 0x2C93, 2, 0x2C95, 0x2C95, 2, 0x2C97, 0x2C97, 2, 0x2C99, 0x2C99, 2, + 0x2C9B, 0x2C9B, 2, 0x2C9D, 0x2C9D, 2, 0x2C9F, 0x2C9F, 2, 0x2CA1, 0x2CA1, 2, + 0x2CA3, 0x2CA3, 2, 0x2CA5, 0x2CA5, 2, 0x2CA7, 0x2CA7, 2, 0x2CA9, 0x2CA9, 2, + 0x2CAB, 0x2CAB, 2, 0x2CAD, 0x2CAD, 2, 0x2CAF, 0x2CAF, 2, 0x2CB1, 0x2CB1, 2, + 0x2CB3, 0x2CB3, 2, 0x2CB5, 0x2CB5, 2, 0x2CB7, 0x2CB7, 2, 0x2CB9, 0x2CB9, 2, + 0x2CBB, 0x2CBB, 2, 0x2CBD, 0x2CBD, 2, 0x2CBF, 0x2CBF, 2, 0x2CC1, 0x2CC1, 2, + 0x2CC3, 0x2CC3, 2, 0x2CC5, 0x2CC5, 2, 0x2CC7, 0x2CC7, 2, 0x2CC9, 0x2CC9, 2, + 0x2CCB, 0x2CCB, 2, 0x2CCD, 0x2CCD, 2, 0x2CCF, 0x2CCF, 2, 0x2CD1, 0x2CD1, 2, + 0x2CD3, 0x2CD3, 2, 0x2CD5, 0x2CD5, 2, 0x2CD7, 0x2CD7, 2, 0x2CD9, 0x2CD9, 2, + 0x2CDB, 0x2CDB, 2, 0x2CDD, 0x2CDD, 2, 0x2CDF, 0x2CDF, 2, 0x2CE1, 0x2CE1, 2, + 0x2CE3, 0x2CE4, 2, 0x2CEC, 0x2CEC, 2, 0x2CEE, 0x2CEE, 2, 0x2CF3, 0x2CF3, 2, + 0x2D00, 0x2D25, 2, 0x2D27, 0x2D27, 2, 0x2D2D, 0x2D2D, 2, 0xA641, 0xA641, 2, + 0xA643, 0xA643, 2, 0xA645, 0xA645, 2, 0xA647, 0xA647, 2, 0xA649, 0xA649, 2, + 0xA64B, 0xA64B, 2, 0xA64D, 0xA64D, 2, 0xA64F, 0xA64F, 2, 0xA651, 0xA651, 2, + 0xA653, 0xA653, 2, 0xA655, 0xA655, 2, 0xA657, 0xA657, 2, 0xA659, 0xA659, 2, + 0xA65B, 0xA65B, 2, 0xA65D, 0xA65D, 2, 0xA65F, 0xA65F, 2, 0xA661, 0xA661, 2, + 0xA663, 0xA663, 2, 0xA665, 0xA665, 2, 0xA667, 0xA667, 2, 0xA669, 0xA669, 2, + 0xA66B, 0xA66B, 2, 0xA66D, 0xA66D, 2, 0xA681, 0xA681, 2, 0xA683, 0xA683, 2, + 0xA685, 0xA685, 2, 0xA687, 0xA687, 2, 0xA689, 0xA689, 2, 0xA68B, 0xA68B, 2, + 0xA68D, 0xA68D, 2, 0xA68F, 0xA68F, 2, 0xA691, 0xA691, 2, 0xA693, 0xA693, 2, + 0xA695, 0xA695, 2, 0xA697, 0xA697, 2, 0xA699, 0xA699, 2, 0xA69B, 0xA69B, 2, + 0xA723, 0xA723, 2, 0xA725, 0xA725, 2, 0xA727, 0xA727, 2, 0xA729, 0xA729, 2, + 0xA72B, 0xA72B, 2, 0xA72D, 0xA72D, 2, 0xA72F, 0xA731, 2, 0xA733, 0xA733, 2, + 0xA735, 0xA735, 2, 0xA737, 0xA737, 2, 0xA739, 0xA739, 2, 0xA73B, 0xA73B, 2, + 0xA73D, 0xA73D, 2, 0xA73F, 0xA73F, 2, 0xA741, 0xA741, 2, 0xA743, 0xA743, 2, + 0xA745, 0xA745, 2, 0xA747, 0xA747, 2, 0xA749, 0xA749, 2, 0xA74B, 0xA74B, 2, + 0xA74D, 0xA74D, 2, 0xA74F, 0xA74F, 2, 0xA751, 0xA751, 2, 0xA753, 0xA753, 2, + 0xA755, 0xA755, 2, 0xA757, 0xA757, 2, 0xA759, 0xA759, 2, 0xA75B, 0xA75B, 2, + 0xA75D, 0xA75D, 2, 0xA75F, 0xA75F, 2, 0xA761, 0xA761, 2, 0xA763, 0xA763, 2, + 0xA765, 0xA765, 2, 0xA767, 0xA767, 2, 0xA769, 0xA769, 2, 0xA76B, 0xA76B, 2, + 0xA76D, 0xA76D, 2, 0xA76F, 0xA76F, 2, 0xA771, 0xA778, 2, 0xA77A, 0xA77A, 2, + 0xA77C, 0xA77C, 2, 0xA77F, 0xA77F, 2, 0xA781, 0xA781, 2, 0xA783, 0xA783, 2, + 0xA785, 0xA785, 2, 0xA787, 0xA787, 2, 0xA78C, 0xA78C, 2, 0xA78E, 0xA78E, 2, + 0xA791, 0xA791, 2, 0xA793, 0xA795, 2, 0xA797, 0xA797, 2, 0xA799, 0xA799, 2, + 0xA79B, 0xA79B, 2, 0xA79D, 0xA79D, 2, 0xA79F, 0xA79F, 2, 0xA7A1, 0xA7A1, 2, + 0xA7A3, 0xA7A3, 2, 0xA7A5, 0xA7A5, 2, 0xA7A7, 0xA7A7, 2, 0xA7A9, 0xA7A9, 2, + 0xA7AF, 0xA7AF, 2, 0xA7B5, 0xA7B5, 2, 0xA7B7, 0xA7B7, 2, 0xA7B9, 0xA7B9, 2, + 0xA7BB, 0xA7BB, 2, 0xA7BD, 0xA7BD, 2, 0xA7BF, 0xA7BF, 2, 0xA7C1, 0xA7C1, 2, + 0xA7C3, 0xA7C3, 2, 0xA7C8, 0xA7C8, 2, 0xA7CA, 0xA7CA, 2, 0xA7CD, 0xA7CD, 2, + 0xA7CF, 0xA7CF, 2, 0xA7D1, 0xA7D1, 2, 0xA7D3, 0xA7D3, 2, 0xA7D5, 0xA7D5, 2, + }; + } + + private static int[] rangeChunk5() { + return new int[] { + 0xA7D7, 0xA7D7, 2, 0xA7D9, 0xA7D9, 2, 0xA7DB, 0xA7DB, 2, 0xA7F6, 0xA7F6, 2, + 0xA7FA, 0xA7FA, 2, 0xAB30, 0xAB5A, 2, 0xAB60, 0xAB68, 2, 0xAB70, 0xABBF, 2, + 0xFB00, 0xFB06, 2, 0xFB13, 0xFB17, 2, 0xFF41, 0xFF5A, 2, 0x10428, 0x1044F, 2, + 0x104D8, 0x104FB, 2, 0x10597, 0x105A1, 2, 0x105A3, 0x105B1, 2, 0x105B3, 0x105B9, 2, + 0x105BB, 0x105BC, 2, 0x10CC0, 0x10CF2, 2, 0x10D70, 0x10D85, 2, 0x118C0, 0x118DF, 2, + 0x16E60, 0x16E7F, 2, 0x16EBB, 0x16ED3, 2, 0x1D41A, 0x1D433, 2, 0x1D44E, 0x1D454, 2, + 0x1D456, 0x1D467, 2, 0x1D482, 0x1D49B, 2, 0x1D4B6, 0x1D4B9, 2, 0x1D4BB, 0x1D4BB, 2, + 0x1D4BD, 0x1D4C3, 2, 0x1D4C5, 0x1D4CF, 2, 0x1D4EA, 0x1D503, 2, 0x1D51E, 0x1D537, 2, + 0x1D552, 0x1D56B, 2, 0x1D586, 0x1D59F, 2, 0x1D5BA, 0x1D5D3, 2, 0x1D5EE, 0x1D607, 2, + 0x1D622, 0x1D63B, 2, 0x1D656, 0x1D66F, 2, 0x1D68A, 0x1D6A5, 2, 0x1D6C2, 0x1D6DA, 2, + 0x1D6DC, 0x1D6E1, 2, 0x1D6FC, 0x1D714, 2, 0x1D716, 0x1D71B, 2, 0x1D736, 0x1D74E, 2, + 0x1D750, 0x1D755, 2, 0x1D770, 0x1D788, 2, 0x1D78A, 0x1D78F, 2, 0x1D7AA, 0x1D7C2, 2, + 0x1D7C4, 0x1D7C9, 2, 0x1D7CB, 0x1D7CB, 2, 0x1DF00, 0x1DF09, 2, 0x1DF0B, 0x1DF1E, 2, + 0x1DF25, 0x1DF2A, 2, 0x1E922, 0x1E943, 2, 0x1C5, 0x1C5, 3, 0x1C8, 0x1C8, 3, + 0x1CB, 0x1CB, 3, 0x1F2, 0x1F2, 3, 0x1F88, 0x1F8F, 3, 0x1F98, 0x1F9F, 3, + 0x1FA8, 0x1FAF, 3, 0x1FBC, 0x1FBC, 3, 0x1FCC, 0x1FCC, 3, 0x1FFC, 0x1FFC, 3, + 0x2B0, 0x2C1, 4, 0x2C6, 0x2D1, 4, 0x2E0, 0x2E4, 4, 0x2EC, 0x2EC, 4, + 0x2EE, 0x2EE, 4, 0x374, 0x374, 4, 0x37A, 0x37A, 4, 0x559, 0x559, 4, + 0x640, 0x640, 4, 0x6E5, 0x6E6, 4, 0x7F4, 0x7F5, 4, 0x7FA, 0x7FA, 4, + 0x81A, 0x81A, 4, 0x824, 0x824, 4, 0x828, 0x828, 4, 0x8C9, 0x8C9, 4, + 0x971, 0x971, 4, 0xE46, 0xE46, 4, 0xEC6, 0xEC6, 4, 0x10FC, 0x10FC, 4, + 0x17D7, 0x17D7, 4, 0x1843, 0x1843, 4, 0x1AA7, 0x1AA7, 4, 0x1C78, 0x1C7D, 4, + 0x1D2C, 0x1D6A, 4, 0x1D78, 0x1D78, 4, 0x1D9B, 0x1DBF, 4, 0x2071, 0x2071, 4, + 0x207F, 0x207F, 4, 0x2090, 0x209C, 4, 0x2C7C, 0x2C7D, 4, 0x2D6F, 0x2D6F, 4, + 0x2E2F, 0x2E2F, 4, 0x3005, 0x3005, 4, 0x3031, 0x3035, 4, 0x303B, 0x303B, 4, + 0x309D, 0x309E, 4, 0x30FC, 0x30FE, 4, 0xA015, 0xA015, 4, 0xA4F8, 0xA4FD, 4, + 0xA60C, 0xA60C, 4, 0xA67F, 0xA67F, 4, 0xA69C, 0xA69D, 4, 0xA717, 0xA71F, 4, + 0xA770, 0xA770, 4, 0xA788, 0xA788, 4, 0xA7F1, 0xA7F4, 4, 0xA7F8, 0xA7F9, 4, + 0xA9CF, 0xA9CF, 4, 0xA9E6, 0xA9E6, 4, 0xAA70, 0xAA70, 4, 0xAADD, 0xAADD, 4, + 0xAAF3, 0xAAF4, 4, 0xAB5C, 0xAB5F, 4, 0xAB69, 0xAB69, 4, 0xFF70, 0xFF70, 4, + 0xFF9E, 0xFF9F, 4, 0x10780, 0x10785, 4, 0x10787, 0x107B0, 4, 0x107B2, 0x107BA, 4, + 0x10D4E, 0x10D4E, 4, 0x10D6F, 0x10D6F, 4, 0x10EC5, 0x10EC5, 4, 0x11DD9, 0x11DD9, 4, + 0x16B40, 0x16B43, 4, 0x16D40, 0x16D42, 4, 0x16D6B, 0x16D6C, 4, 0x16F93, 0x16F9F, 4, + 0x16FE0, 0x16FE1, 4, 0x16FE3, 0x16FE3, 4, 0x16FF2, 0x16FF3, 4, 0x1AFF0, 0x1AFF3, 4, + 0x1AFF5, 0x1AFFB, 4, 0x1AFFD, 0x1AFFE, 4, 0x1E030, 0x1E06D, 4, 0x1E137, 0x1E13D, 4, + 0x1E4EB, 0x1E4EB, 4, 0x1E6FF, 0x1E6FF, 4, 0x1E94B, 0x1E94B, 4, 0xAA, 0xAA, 5, + 0xBA, 0xBA, 5, 0x1BB, 0x1BB, 5, 0x1C0, 0x1C3, 5, 0x294, 0x295, 5, + 0x5D0, 0x5EA, 5, 0x5EF, 0x5F2, 5, 0x620, 0x63F, 5, 0x641, 0x64A, 5, + 0x66E, 0x66F, 5, 0x671, 0x6D3, 5, 0x6D5, 0x6D5, 5, 0x6EE, 0x6EF, 5, + 0x6FA, 0x6FC, 5, 0x6FF, 0x6FF, 5, 0x710, 0x710, 5, 0x712, 0x72F, 5, + 0x74D, 0x7A5, 5, 0x7B1, 0x7B1, 5, 0x7CA, 0x7EA, 5, 0x800, 0x815, 5, + 0x840, 0x858, 5, 0x860, 0x86A, 5, 0x870, 0x887, 5, 0x889, 0x88F, 5, + 0x8A0, 0x8C8, 5, 0x904, 0x939, 5, 0x93D, 0x93D, 5, 0x950, 0x950, 5, + 0x958, 0x961, 5, 0x972, 0x980, 5, 0x985, 0x98C, 5, 0x98F, 0x990, 5, + 0x993, 0x9A8, 5, 0x9AA, 0x9B0, 5, 0x9B2, 0x9B2, 5, 0x9B6, 0x9B9, 5, + 0x9BD, 0x9BD, 5, 0x9CE, 0x9CE, 5, 0x9DC, 0x9DD, 5, 0x9DF, 0x9E1, 5, + 0x9F0, 0x9F1, 5, 0x9FC, 0x9FC, 5, 0xA05, 0xA0A, 5, 0xA0F, 0xA10, 5, + 0xA13, 0xA28, 5, 0xA2A, 0xA30, 5, 0xA32, 0xA33, 5, 0xA35, 0xA36, 5, + 0xA38, 0xA39, 5, 0xA59, 0xA5C, 5, 0xA5E, 0xA5E, 5, 0xA72, 0xA74, 5, + 0xA85, 0xA8D, 5, 0xA8F, 0xA91, 5, 0xA93, 0xAA8, 5, 0xAAA, 0xAB0, 5, + 0xAB2, 0xAB3, 5, 0xAB5, 0xAB9, 5, 0xABD, 0xABD, 5, 0xAD0, 0xAD0, 5, + 0xAE0, 0xAE1, 5, 0xAF9, 0xAF9, 5, 0xB05, 0xB0C, 5, 0xB0F, 0xB10, 5, + 0xB13, 0xB28, 5, 0xB2A, 0xB30, 5, 0xB32, 0xB33, 5, 0xB35, 0xB39, 5, + 0xB3D, 0xB3D, 5, 0xB5C, 0xB5D, 5, 0xB5F, 0xB61, 5, 0xB71, 0xB71, 5, + 0xB83, 0xB83, 5, 0xB85, 0xB8A, 5, 0xB8E, 0xB90, 5, 0xB92, 0xB95, 5, + 0xB99, 0xB9A, 5, 0xB9C, 0xB9C, 5, 0xB9E, 0xB9F, 5, 0xBA3, 0xBA4, 5, + 0xBA8, 0xBAA, 5, 0xBAE, 0xBB9, 5, 0xBD0, 0xBD0, 5, 0xC05, 0xC0C, 5, + 0xC0E, 0xC10, 5, 0xC12, 0xC28, 5, 0xC2A, 0xC39, 5, 0xC3D, 0xC3D, 5, + 0xC58, 0xC5A, 5, 0xC5C, 0xC5D, 5, 0xC60, 0xC61, 5, 0xC80, 0xC80, 5, + 0xC85, 0xC8C, 5, 0xC8E, 0xC90, 5, 0xC92, 0xCA8, 5, 0xCAA, 0xCB3, 5, + 0xCB5, 0xCB9, 5, 0xCBD, 0xCBD, 5, 0xCDC, 0xCDE, 5, 0xCE0, 0xCE1, 5, + 0xCF1, 0xCF2, 5, 0xD04, 0xD0C, 5, 0xD0E, 0xD10, 5, 0xD12, 0xD3A, 5, + 0xD3D, 0xD3D, 5, 0xD4E, 0xD4E, 5, 0xD54, 0xD56, 5, 0xD5F, 0xD61, 5, + 0xD7A, 0xD7F, 5, 0xD85, 0xD96, 5, 0xD9A, 0xDB1, 5, 0xDB3, 0xDBB, 5, + 0xDBD, 0xDBD, 5, 0xDC0, 0xDC6, 5, 0xE01, 0xE30, 5, 0xE32, 0xE33, 5, + 0xE40, 0xE45, 5, 0xE81, 0xE82, 5, 0xE84, 0xE84, 5, 0xE86, 0xE8A, 5, + 0xE8C, 0xEA3, 5, 0xEA5, 0xEA5, 5, 0xEA7, 0xEB0, 5, 0xEB2, 0xEB3, 5, + 0xEBD, 0xEBD, 5, 0xEC0, 0xEC4, 5, 0xEDC, 0xEDF, 5, 0xF00, 0xF00, 5, + 0xF40, 0xF47, 5, 0xF49, 0xF6C, 5, 0xF88, 0xF8C, 5, 0x1000, 0x102A, 5, + 0x103F, 0x103F, 5, 0x1050, 0x1055, 5, 0x105A, 0x105D, 5, 0x1061, 0x1061, 5, + 0x1065, 0x1066, 5, 0x106E, 0x1070, 5, 0x1075, 0x1081, 5, 0x108E, 0x108E, 5, + 0x1100, 0x1248, 5, 0x124A, 0x124D, 5, 0x1250, 0x1256, 5, 0x1258, 0x1258, 5, + 0x125A, 0x125D, 5, 0x1260, 0x1288, 5, 0x128A, 0x128D, 5, 0x1290, 0x12B0, 5, + 0x12B2, 0x12B5, 5, 0x12B8, 0x12BE, 5, 0x12C0, 0x12C0, 5, 0x12C2, 0x12C5, 5, + 0x12C8, 0x12D6, 5, 0x12D8, 0x1310, 5, 0x1312, 0x1315, 5, 0x1318, 0x135A, 5, + 0x1380, 0x138F, 5, 0x1401, 0x166C, 5, 0x166F, 0x167F, 5, 0x1681, 0x169A, 5, + 0x16A0, 0x16EA, 5, 0x16F1, 0x16F8, 5, 0x1700, 0x1711, 5, 0x171F, 0x1731, 5, + 0x1740, 0x1751, 5, 0x1760, 0x176C, 5, 0x176E, 0x1770, 5, 0x1780, 0x17B3, 5, + 0x17DC, 0x17DC, 5, 0x1820, 0x1842, 5, 0x1844, 0x1878, 5, 0x1880, 0x1884, 5, + 0x1887, 0x18A8, 5, 0x18AA, 0x18AA, 5, 0x18B0, 0x18F5, 5, 0x1900, 0x191E, 5, + 0x1950, 0x196D, 5, 0x1970, 0x1974, 5, 0x1980, 0x19AB, 5, 0x19B0, 0x19C9, 5, + 0x1A00, 0x1A16, 5, 0x1A20, 0x1A54, 5, 0x1B05, 0x1B33, 5, 0x1B45, 0x1B4C, 5, + 0x1B83, 0x1BA0, 5, 0x1BAE, 0x1BAF, 5, 0x1BBA, 0x1BE5, 5, 0x1C00, 0x1C23, 5, + 0x1C4D, 0x1C4F, 5, 0x1C5A, 0x1C77, 5, 0x1CE9, 0x1CEC, 5, 0x1CEE, 0x1CF3, 5, + 0x1CF5, 0x1CF6, 5, 0x1CFA, 0x1CFA, 5, 0x2135, 0x2138, 5, 0x2D30, 0x2D67, 5, + 0x2D80, 0x2D96, 5, 0x2DA0, 0x2DA6, 5, 0x2DA8, 0x2DAE, 5, 0x2DB0, 0x2DB6, 5, + 0x2DB8, 0x2DBE, 5, 0x2DC0, 0x2DC6, 5, 0x2DC8, 0x2DCE, 5, 0x2DD0, 0x2DD6, 5, + 0x2DD8, 0x2DDE, 5, 0x3006, 0x3006, 5, 0x303C, 0x303C, 5, 0x3041, 0x3096, 5, + 0x309F, 0x309F, 5, 0x30A1, 0x30FA, 5, 0x30FF, 0x30FF, 5, 0x3105, 0x312F, 5, + 0x3131, 0x318E, 5, 0x31A0, 0x31BF, 5, 0x31F0, 0x31FF, 5, 0x3400, 0x4DBF, 5, + 0x4E00, 0xA014, 5, 0xA016, 0xA48C, 5, 0xA4D0, 0xA4F7, 5, 0xA500, 0xA60B, 5, + 0xA610, 0xA61F, 5, 0xA62A, 0xA62B, 5, 0xA66E, 0xA66E, 5, 0xA6A0, 0xA6E5, 5, + 0xA78F, 0xA78F, 5, 0xA7F7, 0xA7F7, 5, 0xA7FB, 0xA801, 5, 0xA803, 0xA805, 5, + 0xA807, 0xA80A, 5, 0xA80C, 0xA822, 5, 0xA840, 0xA873, 5, 0xA882, 0xA8B3, 5, + 0xA8F2, 0xA8F7, 5, 0xA8FB, 0xA8FB, 5, 0xA8FD, 0xA8FE, 5, 0xA90A, 0xA925, 5, + 0xA930, 0xA946, 5, 0xA960, 0xA97C, 5, 0xA984, 0xA9B2, 5, 0xA9E0, 0xA9E4, 5, + 0xA9E7, 0xA9EF, 5, 0xA9FA, 0xA9FE, 5, 0xAA00, 0xAA28, 5, 0xAA40, 0xAA42, 5, + 0xAA44, 0xAA4B, 5, 0xAA60, 0xAA6F, 5, 0xAA71, 0xAA76, 5, 0xAA7A, 0xAA7A, 5, + 0xAA7E, 0xAAAF, 5, 0xAAB1, 0xAAB1, 5, 0xAAB5, 0xAAB6, 5, 0xAAB9, 0xAABD, 5, + 0xAAC0, 0xAAC0, 5, 0xAAC2, 0xAAC2, 5, 0xAADB, 0xAADC, 5, 0xAAE0, 0xAAEA, 5, + }; + } + + private static int[] rangeChunk6() { + return new int[] { + 0xAAF2, 0xAAF2, 5, 0xAB01, 0xAB06, 5, 0xAB09, 0xAB0E, 5, 0xAB11, 0xAB16, 5, + 0xAB20, 0xAB26, 5, 0xAB28, 0xAB2E, 5, 0xABC0, 0xABE2, 5, 0xAC00, 0xD7A3, 5, + 0xD7B0, 0xD7C6, 5, 0xD7CB, 0xD7FB, 5, 0xF900, 0xFA6D, 5, 0xFA70, 0xFAD9, 5, + 0xFB1D, 0xFB1D, 5, 0xFB1F, 0xFB28, 5, 0xFB2A, 0xFB36, 5, 0xFB38, 0xFB3C, 5, + 0xFB3E, 0xFB3E, 5, 0xFB40, 0xFB41, 5, 0xFB43, 0xFB44, 5, 0xFB46, 0xFBB1, 5, + 0xFBD3, 0xFD3D, 5, 0xFD50, 0xFD8F, 5, 0xFD92, 0xFDC7, 5, 0xFDF0, 0xFDFB, 5, + 0xFE70, 0xFE74, 5, 0xFE76, 0xFEFC, 5, 0xFF66, 0xFF6F, 5, 0xFF71, 0xFF9D, 5, + 0xFFA0, 0xFFBE, 5, 0xFFC2, 0xFFC7, 5, 0xFFCA, 0xFFCF, 5, 0xFFD2, 0xFFD7, 5, + 0xFFDA, 0xFFDC, 5, 0x10000, 0x1000B, 5, 0x1000D, 0x10026, 5, 0x10028, 0x1003A, 5, + 0x1003C, 0x1003D, 5, 0x1003F, 0x1004D, 5, 0x10050, 0x1005D, 5, 0x10080, 0x100FA, 5, + 0x10280, 0x1029C, 5, 0x102A0, 0x102D0, 5, 0x10300, 0x1031F, 5, 0x1032D, 0x10340, 5, + 0x10342, 0x10349, 5, 0x10350, 0x10375, 5, 0x10380, 0x1039D, 5, 0x103A0, 0x103C3, 5, + 0x103C8, 0x103CF, 5, 0x10450, 0x1049D, 5, 0x10500, 0x10527, 5, 0x10530, 0x10563, 5, + 0x105C0, 0x105F3, 5, 0x10600, 0x10736, 5, 0x10740, 0x10755, 5, 0x10760, 0x10767, 5, + 0x10800, 0x10805, 5, 0x10808, 0x10808, 5, 0x1080A, 0x10835, 5, 0x10837, 0x10838, 5, + 0x1083C, 0x1083C, 5, 0x1083F, 0x10855, 5, 0x10860, 0x10876, 5, 0x10880, 0x1089E, 5, + 0x108E0, 0x108F2, 5, 0x108F4, 0x108F5, 5, 0x10900, 0x10915, 5, 0x10920, 0x10939, 5, + 0x10940, 0x10959, 5, 0x10980, 0x109B7, 5, 0x109BE, 0x109BF, 5, 0x10A00, 0x10A00, 5, + 0x10A10, 0x10A13, 5, 0x10A15, 0x10A17, 5, 0x10A19, 0x10A35, 5, 0x10A60, 0x10A7C, 5, + 0x10A80, 0x10A9C, 5, 0x10AC0, 0x10AC7, 5, 0x10AC9, 0x10AE4, 5, 0x10B00, 0x10B35, 5, + 0x10B40, 0x10B55, 5, 0x10B60, 0x10B72, 5, 0x10B80, 0x10B91, 5, 0x10C00, 0x10C48, 5, + 0x10D00, 0x10D23, 5, 0x10D4A, 0x10D4D, 5, 0x10D4F, 0x10D4F, 5, 0x10E80, 0x10EA9, 5, + 0x10EB0, 0x10EB1, 5, 0x10EC2, 0x10EC4, 5, 0x10EC6, 0x10EC7, 5, 0x10F00, 0x10F1C, 5, + 0x10F27, 0x10F27, 5, 0x10F30, 0x10F45, 5, 0x10F70, 0x10F81, 5, 0x10FB0, 0x10FC4, 5, + 0x10FE0, 0x10FF6, 5, 0x11003, 0x11037, 5, 0x11071, 0x11072, 5, 0x11075, 0x11075, 5, + 0x11083, 0x110AF, 5, 0x110D0, 0x110E8, 5, 0x11103, 0x11126, 5, 0x11144, 0x11144, 5, + 0x11147, 0x11147, 5, 0x11150, 0x11172, 5, 0x11176, 0x11176, 5, 0x11183, 0x111B2, 5, + 0x111C1, 0x111C4, 5, 0x111DA, 0x111DA, 5, 0x111DC, 0x111DC, 5, 0x11200, 0x11211, 5, + 0x11213, 0x1122B, 5, 0x1123F, 0x11240, 5, 0x11280, 0x11286, 5, 0x11288, 0x11288, 5, + 0x1128A, 0x1128D, 5, 0x1128F, 0x1129D, 5, 0x1129F, 0x112A8, 5, 0x112B0, 0x112DE, 5, + 0x11305, 0x1130C, 5, 0x1130F, 0x11310, 5, 0x11313, 0x11328, 5, 0x1132A, 0x11330, 5, + 0x11332, 0x11333, 5, 0x11335, 0x11339, 5, 0x1133D, 0x1133D, 5, 0x11350, 0x11350, 5, + 0x1135D, 0x11361, 5, 0x11380, 0x11389, 5, 0x1138B, 0x1138B, 5, 0x1138E, 0x1138E, 5, + 0x11390, 0x113B5, 5, 0x113B7, 0x113B7, 5, 0x113D1, 0x113D1, 5, 0x113D3, 0x113D3, 5, + 0x11400, 0x11434, 5, 0x11447, 0x1144A, 5, 0x1145F, 0x11461, 5, 0x11480, 0x114AF, 5, + 0x114C4, 0x114C5, 5, 0x114C7, 0x114C7, 5, 0x11580, 0x115AE, 5, 0x115D8, 0x115DB, 5, + 0x11600, 0x1162F, 5, 0x11644, 0x11644, 5, 0x11680, 0x116AA, 5, 0x116B8, 0x116B8, 5, + 0x11700, 0x1171A, 5, 0x11740, 0x11746, 5, 0x11800, 0x1182B, 5, 0x118FF, 0x11906, 5, + 0x11909, 0x11909, 5, 0x1190C, 0x11913, 5, 0x11915, 0x11916, 5, 0x11918, 0x1192F, 5, + 0x1193F, 0x1193F, 5, 0x11941, 0x11941, 5, 0x119A0, 0x119A7, 5, 0x119AA, 0x119D0, 5, + 0x119E1, 0x119E1, 5, 0x119E3, 0x119E3, 5, 0x11A00, 0x11A00, 5, 0x11A0B, 0x11A32, 5, + 0x11A3A, 0x11A3A, 5, 0x11A50, 0x11A50, 5, 0x11A5C, 0x11A89, 5, 0x11A9D, 0x11A9D, 5, + 0x11AB0, 0x11AF8, 5, 0x11BC0, 0x11BE0, 5, 0x11C00, 0x11C08, 5, 0x11C0A, 0x11C2E, 5, + 0x11C40, 0x11C40, 5, 0x11C72, 0x11C8F, 5, 0x11D00, 0x11D06, 5, 0x11D08, 0x11D09, 5, + 0x11D0B, 0x11D30, 5, 0x11D46, 0x11D46, 5, 0x11D60, 0x11D65, 5, 0x11D67, 0x11D68, 5, + 0x11D6A, 0x11D89, 5, 0x11D98, 0x11D98, 5, 0x11DB0, 0x11DD8, 5, 0x11DDA, 0x11DDB, 5, + 0x11EE0, 0x11EF2, 5, 0x11F02, 0x11F02, 5, 0x11F04, 0x11F10, 5, 0x11F12, 0x11F33, 5, + 0x11FB0, 0x11FB0, 5, 0x12000, 0x12399, 5, 0x12480, 0x12543, 5, 0x12F90, 0x12FF0, 5, + 0x13000, 0x1342F, 5, 0x13441, 0x13446, 5, 0x13460, 0x143FA, 5, 0x14400, 0x14646, 5, + 0x16100, 0x1611D, 5, 0x16800, 0x16A38, 5, 0x16A40, 0x16A5E, 5, 0x16A70, 0x16ABE, 5, + 0x16AD0, 0x16AED, 5, 0x16B00, 0x16B2F, 5, 0x16B63, 0x16B77, 5, 0x16B7D, 0x16B8F, 5, + 0x16D43, 0x16D6A, 5, 0x16F00, 0x16F4A, 5, 0x16F50, 0x16F50, 5, 0x17000, 0x18CD5, 5, + 0x18CFF, 0x18D1E, 5, 0x18D80, 0x18DF2, 5, 0x1B000, 0x1B122, 5, 0x1B132, 0x1B132, 5, + 0x1B150, 0x1B152, 5, 0x1B155, 0x1B155, 5, 0x1B164, 0x1B167, 5, 0x1B170, 0x1B2FB, 5, + 0x1BC00, 0x1BC6A, 5, 0x1BC70, 0x1BC7C, 5, 0x1BC80, 0x1BC88, 5, 0x1BC90, 0x1BC99, 5, + 0x1DF0A, 0x1DF0A, 5, 0x1E100, 0x1E12C, 5, 0x1E14E, 0x1E14E, 5, 0x1E290, 0x1E2AD, 5, + 0x1E2C0, 0x1E2EB, 5, 0x1E4D0, 0x1E4EA, 5, 0x1E5D0, 0x1E5ED, 5, 0x1E5F0, 0x1E5F0, 5, + 0x1E6C0, 0x1E6DE, 5, 0x1E6E0, 0x1E6E2, 5, 0x1E6E4, 0x1E6E5, 5, 0x1E6E7, 0x1E6ED, 5, + 0x1E6F0, 0x1E6F4, 5, 0x1E6FE, 0x1E6FE, 5, 0x1E7E0, 0x1E7E6, 5, 0x1E7E8, 0x1E7EB, 5, + 0x1E7ED, 0x1E7EE, 5, 0x1E7F0, 0x1E7FE, 5, 0x1E800, 0x1E8C4, 5, 0x1EE00, 0x1EE03, 5, + 0x1EE05, 0x1EE1F, 5, 0x1EE21, 0x1EE22, 5, 0x1EE24, 0x1EE24, 5, 0x1EE27, 0x1EE27, 5, + 0x1EE29, 0x1EE32, 5, 0x1EE34, 0x1EE37, 5, 0x1EE39, 0x1EE39, 5, 0x1EE3B, 0x1EE3B, 5, + 0x1EE42, 0x1EE42, 5, 0x1EE47, 0x1EE47, 5, 0x1EE49, 0x1EE49, 5, 0x1EE4B, 0x1EE4B, 5, + 0x1EE4D, 0x1EE4F, 5, 0x1EE51, 0x1EE52, 5, 0x1EE54, 0x1EE54, 5, 0x1EE57, 0x1EE57, 5, + 0x1EE59, 0x1EE59, 5, 0x1EE5B, 0x1EE5B, 5, 0x1EE5D, 0x1EE5D, 5, 0x1EE5F, 0x1EE5F, 5, + 0x1EE61, 0x1EE62, 5, 0x1EE64, 0x1EE64, 5, 0x1EE67, 0x1EE6A, 5, 0x1EE6C, 0x1EE72, 5, + 0x1EE74, 0x1EE77, 5, 0x1EE79, 0x1EE7C, 5, 0x1EE7E, 0x1EE7E, 5, 0x1EE80, 0x1EE89, 5, + 0x1EE8B, 0x1EE9B, 5, 0x1EEA1, 0x1EEA3, 5, 0x1EEA5, 0x1EEA9, 5, 0x1EEAB, 0x1EEBB, 5, + 0x20000, 0x2A6DF, 5, 0x2A700, 0x2B81D, 5, 0x2B820, 0x2CEAD, 5, 0x2CEB0, 0x2EBE0, 5, + 0x2EBF0, 0x2EE5D, 5, 0x2F800, 0x2FA1D, 5, 0x30000, 0x3134A, 5, 0x31350, 0x33479, 5, + 0x300, 0x36F, 6, 0x483, 0x487, 6, 0x591, 0x5BD, 6, 0x5BF, 0x5BF, 6, + 0x5C1, 0x5C2, 6, 0x5C4, 0x5C5, 6, 0x5C7, 0x5C7, 6, 0x610, 0x61A, 6, + 0x64B, 0x65F, 6, 0x670, 0x670, 6, 0x6D6, 0x6DC, 6, 0x6DF, 0x6E4, 6, + 0x6E7, 0x6E8, 6, 0x6EA, 0x6ED, 6, 0x711, 0x711, 6, 0x730, 0x74A, 6, + 0x7A6, 0x7B0, 6, 0x7EB, 0x7F3, 6, 0x7FD, 0x7FD, 6, 0x816, 0x819, 6, + 0x81B, 0x823, 6, 0x825, 0x827, 6, 0x829, 0x82D, 6, 0x859, 0x85B, 6, + 0x897, 0x89F, 6, 0x8CA, 0x8E1, 6, 0x8E3, 0x902, 6, 0x93A, 0x93A, 6, + 0x93C, 0x93C, 6, 0x941, 0x948, 6, 0x94D, 0x94D, 6, 0x951, 0x957, 6, + 0x962, 0x963, 6, 0x981, 0x981, 6, 0x9BC, 0x9BC, 6, 0x9C1, 0x9C4, 6, + 0x9CD, 0x9CD, 6, 0x9E2, 0x9E3, 6, 0x9FE, 0x9FE, 6, 0xA01, 0xA02, 6, + 0xA3C, 0xA3C, 6, 0xA41, 0xA42, 6, 0xA47, 0xA48, 6, 0xA4B, 0xA4D, 6, + 0xA51, 0xA51, 6, 0xA70, 0xA71, 6, 0xA75, 0xA75, 6, 0xA81, 0xA82, 6, + 0xABC, 0xABC, 6, 0xAC1, 0xAC5, 6, 0xAC7, 0xAC8, 6, 0xACD, 0xACD, 6, + 0xAE2, 0xAE3, 6, 0xAFA, 0xAFF, 6, 0xB01, 0xB01, 6, 0xB3C, 0xB3C, 6, + 0xB3F, 0xB3F, 6, 0xB41, 0xB44, 6, 0xB4D, 0xB4D, 6, 0xB55, 0xB56, 6, + 0xB62, 0xB63, 6, 0xB82, 0xB82, 6, 0xBC0, 0xBC0, 6, 0xBCD, 0xBCD, 6, + 0xC00, 0xC00, 6, 0xC04, 0xC04, 6, 0xC3C, 0xC3C, 6, 0xC3E, 0xC40, 6, + 0xC46, 0xC48, 6, 0xC4A, 0xC4D, 6, 0xC55, 0xC56, 6, 0xC62, 0xC63, 6, + 0xC81, 0xC81, 6, 0xCBC, 0xCBC, 6, 0xCBF, 0xCBF, 6, 0xCC6, 0xCC6, 6, + 0xCCC, 0xCCD, 6, 0xCE2, 0xCE3, 6, 0xD00, 0xD01, 6, 0xD3B, 0xD3C, 6, + 0xD41, 0xD44, 6, 0xD4D, 0xD4D, 6, 0xD62, 0xD63, 6, 0xD81, 0xD81, 6, + 0xDCA, 0xDCA, 6, 0xDD2, 0xDD4, 6, 0xDD6, 0xDD6, 6, 0xE31, 0xE31, 6, + 0xE34, 0xE3A, 6, 0xE47, 0xE4E, 6, 0xEB1, 0xEB1, 6, 0xEB4, 0xEBC, 6, + 0xEC8, 0xECE, 6, 0xF18, 0xF19, 6, 0xF35, 0xF35, 6, 0xF37, 0xF37, 6, + 0xF39, 0xF39, 6, 0xF71, 0xF7E, 6, 0xF80, 0xF84, 6, 0xF86, 0xF87, 6, + 0xF8D, 0xF97, 6, 0xF99, 0xFBC, 6, 0xFC6, 0xFC6, 6, 0x102D, 0x1030, 6, + 0x1032, 0x1037, 6, 0x1039, 0x103A, 6, 0x103D, 0x103E, 6, 0x1058, 0x1059, 6, + 0x105E, 0x1060, 6, 0x1071, 0x1074, 6, 0x1082, 0x1082, 6, 0x1085, 0x1086, 6, + 0x108D, 0x108D, 6, 0x109D, 0x109D, 6, 0x135D, 0x135F, 6, 0x1712, 0x1714, 6, + 0x1732, 0x1733, 6, 0x1752, 0x1753, 6, 0x1772, 0x1773, 6, 0x17B4, 0x17B5, 6, + }; + } + + private static int[] rangeChunk7() { + return new int[] { + 0x17B7, 0x17BD, 6, 0x17C6, 0x17C6, 6, 0x17C9, 0x17D3, 6, 0x17DD, 0x17DD, 6, + 0x180B, 0x180D, 6, 0x180F, 0x180F, 6, 0x1885, 0x1886, 6, 0x18A9, 0x18A9, 6, + 0x1920, 0x1922, 6, 0x1927, 0x1928, 6, 0x1932, 0x1932, 6, 0x1939, 0x193B, 6, + 0x1A17, 0x1A18, 6, 0x1A1B, 0x1A1B, 6, 0x1A56, 0x1A56, 6, 0x1A58, 0x1A5E, 6, + 0x1A60, 0x1A60, 6, 0x1A62, 0x1A62, 6, 0x1A65, 0x1A6C, 6, 0x1A73, 0x1A7C, 6, + 0x1A7F, 0x1A7F, 6, 0x1AB0, 0x1ABD, 6, 0x1ABF, 0x1ADD, 6, 0x1AE0, 0x1AEB, 6, + 0x1B00, 0x1B03, 6, 0x1B34, 0x1B34, 6, 0x1B36, 0x1B3A, 6, 0x1B3C, 0x1B3C, 6, + 0x1B42, 0x1B42, 6, 0x1B6B, 0x1B73, 6, 0x1B80, 0x1B81, 6, 0x1BA2, 0x1BA5, 6, + 0x1BA8, 0x1BA9, 6, 0x1BAB, 0x1BAD, 6, 0x1BE6, 0x1BE6, 6, 0x1BE8, 0x1BE9, 6, + 0x1BED, 0x1BED, 6, 0x1BEF, 0x1BF1, 6, 0x1C2C, 0x1C33, 6, 0x1C36, 0x1C37, 6, + 0x1CD0, 0x1CD2, 6, 0x1CD4, 0x1CE0, 6, 0x1CE2, 0x1CE8, 6, 0x1CED, 0x1CED, 6, + 0x1CF4, 0x1CF4, 6, 0x1CF8, 0x1CF9, 6, 0x1DC0, 0x1DFF, 6, 0x20D0, 0x20DC, 6, + 0x20E1, 0x20E1, 6, 0x20E5, 0x20F0, 6, 0x2CEF, 0x2CF1, 6, 0x2D7F, 0x2D7F, 6, + 0x2DE0, 0x2DFF, 6, 0x302A, 0x302D, 6, 0x3099, 0x309A, 6, 0xA66F, 0xA66F, 6, + 0xA674, 0xA67D, 6, 0xA69E, 0xA69F, 6, 0xA6F0, 0xA6F1, 6, 0xA802, 0xA802, 6, + 0xA806, 0xA806, 6, 0xA80B, 0xA80B, 6, 0xA825, 0xA826, 6, 0xA82C, 0xA82C, 6, + 0xA8C4, 0xA8C5, 6, 0xA8E0, 0xA8F1, 6, 0xA8FF, 0xA8FF, 6, 0xA926, 0xA92D, 6, + 0xA947, 0xA951, 6, 0xA980, 0xA982, 6, 0xA9B3, 0xA9B3, 6, 0xA9B6, 0xA9B9, 6, + 0xA9BC, 0xA9BD, 6, 0xA9E5, 0xA9E5, 6, 0xAA29, 0xAA2E, 6, 0xAA31, 0xAA32, 6, + 0xAA35, 0xAA36, 6, 0xAA43, 0xAA43, 6, 0xAA4C, 0xAA4C, 6, 0xAA7C, 0xAA7C, 6, + 0xAAB0, 0xAAB0, 6, 0xAAB2, 0xAAB4, 6, 0xAAB7, 0xAAB8, 6, 0xAABE, 0xAABF, 6, + 0xAAC1, 0xAAC1, 6, 0xAAEC, 0xAAED, 6, 0xAAF6, 0xAAF6, 6, 0xABE5, 0xABE5, 6, + 0xABE8, 0xABE8, 6, 0xABED, 0xABED, 6, 0xFB1E, 0xFB1E, 6, 0xFE00, 0xFE0F, 6, + 0xFE20, 0xFE2F, 6, 0x101FD, 0x101FD, 6, 0x102E0, 0x102E0, 6, 0x10376, 0x1037A, 6, + 0x10A01, 0x10A03, 6, 0x10A05, 0x10A06, 6, 0x10A0C, 0x10A0F, 6, 0x10A38, 0x10A3A, 6, + 0x10A3F, 0x10A3F, 6, 0x10AE5, 0x10AE6, 6, 0x10D24, 0x10D27, 6, 0x10D69, 0x10D6D, 6, + 0x10EAB, 0x10EAC, 6, 0x10EFA, 0x10EFF, 6, 0x10F46, 0x10F50, 6, 0x10F82, 0x10F85, 6, + 0x11001, 0x11001, 6, 0x11038, 0x11046, 6, 0x11070, 0x11070, 6, 0x11073, 0x11074, 6, + 0x1107F, 0x11081, 6, 0x110B3, 0x110B6, 6, 0x110B9, 0x110BA, 6, 0x110C2, 0x110C2, 6, + 0x11100, 0x11102, 6, 0x11127, 0x1112B, 6, 0x1112D, 0x11134, 6, 0x11173, 0x11173, 6, + 0x11180, 0x11181, 6, 0x111B6, 0x111BE, 6, 0x111C9, 0x111CC, 6, 0x111CF, 0x111CF, 6, + 0x1122F, 0x11231, 6, 0x11234, 0x11234, 6, 0x11236, 0x11237, 6, 0x1123E, 0x1123E, 6, + 0x11241, 0x11241, 6, 0x112DF, 0x112DF, 6, 0x112E3, 0x112EA, 6, 0x11300, 0x11301, 6, + 0x1133B, 0x1133C, 6, 0x11340, 0x11340, 6, 0x11366, 0x1136C, 6, 0x11370, 0x11374, 6, + 0x113BB, 0x113C0, 6, 0x113CE, 0x113CE, 6, 0x113D0, 0x113D0, 6, 0x113D2, 0x113D2, 6, + 0x113E1, 0x113E2, 6, 0x11438, 0x1143F, 6, 0x11442, 0x11444, 6, 0x11446, 0x11446, 6, + 0x1145E, 0x1145E, 6, 0x114B3, 0x114B8, 6, 0x114BA, 0x114BA, 6, 0x114BF, 0x114C0, 6, + 0x114C2, 0x114C3, 6, 0x115B2, 0x115B5, 6, 0x115BC, 0x115BD, 6, 0x115BF, 0x115C0, 6, + 0x115DC, 0x115DD, 6, 0x11633, 0x1163A, 6, 0x1163D, 0x1163D, 6, 0x1163F, 0x11640, 6, + 0x116AB, 0x116AB, 6, 0x116AD, 0x116AD, 6, 0x116B0, 0x116B5, 6, 0x116B7, 0x116B7, 6, + 0x1171D, 0x1171D, 6, 0x1171F, 0x1171F, 6, 0x11722, 0x11725, 6, 0x11727, 0x1172B, 6, + 0x1182F, 0x11837, 6, 0x11839, 0x1183A, 6, 0x1193B, 0x1193C, 6, 0x1193E, 0x1193E, 6, + 0x11943, 0x11943, 6, 0x119D4, 0x119D7, 6, 0x119DA, 0x119DB, 6, 0x119E0, 0x119E0, 6, + 0x11A01, 0x11A0A, 6, 0x11A33, 0x11A38, 6, 0x11A3B, 0x11A3E, 6, 0x11A47, 0x11A47, 6, + 0x11A51, 0x11A56, 6, 0x11A59, 0x11A5B, 6, 0x11A8A, 0x11A96, 6, 0x11A98, 0x11A99, 6, + 0x11B60, 0x11B60, 6, 0x11B62, 0x11B64, 6, 0x11B66, 0x11B66, 6, 0x11C30, 0x11C36, 6, + 0x11C38, 0x11C3D, 6, 0x11C3F, 0x11C3F, 6, 0x11C92, 0x11CA7, 6, 0x11CAA, 0x11CB0, 6, + 0x11CB2, 0x11CB3, 6, 0x11CB5, 0x11CB6, 6, 0x11D31, 0x11D36, 6, 0x11D3A, 0x11D3A, 6, + 0x11D3C, 0x11D3D, 6, 0x11D3F, 0x11D45, 6, 0x11D47, 0x11D47, 6, 0x11D90, 0x11D91, 6, + 0x11D95, 0x11D95, 6, 0x11D97, 0x11D97, 6, 0x11EF3, 0x11EF4, 6, 0x11F00, 0x11F01, 6, + 0x11F36, 0x11F3A, 6, 0x11F40, 0x11F40, 6, 0x11F42, 0x11F42, 6, 0x11F5A, 0x11F5A, 6, + 0x13440, 0x13440, 6, 0x13447, 0x13455, 6, 0x1611E, 0x16129, 6, 0x1612D, 0x1612F, 6, + 0x16AF0, 0x16AF4, 6, 0x16B30, 0x16B36, 6, 0x16F4F, 0x16F4F, 6, 0x16F8F, 0x16F92, 6, + 0x16FE4, 0x16FE4, 6, 0x1BC9D, 0x1BC9E, 6, 0x1CF00, 0x1CF2D, 6, 0x1CF30, 0x1CF46, 6, + 0x1D167, 0x1D169, 6, 0x1D17B, 0x1D182, 6, 0x1D185, 0x1D18B, 6, 0x1D1AA, 0x1D1AD, 6, + 0x1D242, 0x1D244, 6, 0x1DA00, 0x1DA36, 6, 0x1DA3B, 0x1DA6C, 6, 0x1DA75, 0x1DA75, 6, + 0x1DA84, 0x1DA84, 6, 0x1DA9B, 0x1DA9F, 6, 0x1DAA1, 0x1DAAF, 6, 0x1E000, 0x1E006, 6, + 0x1E008, 0x1E018, 6, 0x1E01B, 0x1E021, 6, 0x1E023, 0x1E024, 6, 0x1E026, 0x1E02A, 6, + 0x1E08F, 0x1E08F, 6, 0x1E130, 0x1E136, 6, 0x1E2AE, 0x1E2AE, 6, 0x1E2EC, 0x1E2EF, 6, + 0x1E4EC, 0x1E4EF, 6, 0x1E5EE, 0x1E5EF, 6, 0x1E6E3, 0x1E6E3, 6, 0x1E6E6, 0x1E6E6, 6, + 0x1E6EE, 0x1E6EF, 6, 0x1E6F5, 0x1E6F5, 6, 0x1E8D0, 0x1E8D6, 6, 0x1E944, 0x1E94A, 6, + 0xE0100, 0xE01EF, 6, 0x488, 0x489, 7, 0x1ABE, 0x1ABE, 7, 0x20DD, 0x20E0, 7, + 0x20E2, 0x20E4, 7, 0xA670, 0xA672, 7, 0x903, 0x903, 8, 0x93B, 0x93B, 8, + 0x93E, 0x940, 8, 0x949, 0x94C, 8, 0x94E, 0x94F, 8, 0x982, 0x983, 8, + 0x9BE, 0x9C0, 8, 0x9C7, 0x9C8, 8, 0x9CB, 0x9CC, 8, 0x9D7, 0x9D7, 8, + 0xA03, 0xA03, 8, 0xA3E, 0xA40, 8, 0xA83, 0xA83, 8, 0xABE, 0xAC0, 8, + 0xAC9, 0xAC9, 8, 0xACB, 0xACC, 8, 0xB02, 0xB03, 8, 0xB3E, 0xB3E, 8, + 0xB40, 0xB40, 8, 0xB47, 0xB48, 8, 0xB4B, 0xB4C, 8, 0xB57, 0xB57, 8, + 0xBBE, 0xBBF, 8, 0xBC1, 0xBC2, 8, 0xBC6, 0xBC8, 8, 0xBCA, 0xBCC, 8, + 0xBD7, 0xBD7, 8, 0xC01, 0xC03, 8, 0xC41, 0xC44, 8, 0xC82, 0xC83, 8, + 0xCBE, 0xCBE, 8, 0xCC0, 0xCC4, 8, 0xCC7, 0xCC8, 8, 0xCCA, 0xCCB, 8, + 0xCD5, 0xCD6, 8, 0xCF3, 0xCF3, 8, 0xD02, 0xD03, 8, 0xD3E, 0xD40, 8, + 0xD46, 0xD48, 8, 0xD4A, 0xD4C, 8, 0xD57, 0xD57, 8, 0xD82, 0xD83, 8, + 0xDCF, 0xDD1, 8, 0xDD8, 0xDDF, 8, 0xDF2, 0xDF3, 8, 0xF3E, 0xF3F, 8, + 0xF7F, 0xF7F, 8, 0x102B, 0x102C, 8, 0x1031, 0x1031, 8, 0x1038, 0x1038, 8, + 0x103B, 0x103C, 8, 0x1056, 0x1057, 8, 0x1062, 0x1064, 8, 0x1067, 0x106D, 8, + 0x1083, 0x1084, 8, 0x1087, 0x108C, 8, 0x108F, 0x108F, 8, 0x109A, 0x109C, 8, + 0x1715, 0x1715, 8, 0x1734, 0x1734, 8, 0x17B6, 0x17B6, 8, 0x17BE, 0x17C5, 8, + 0x17C7, 0x17C8, 8, 0x1923, 0x1926, 8, 0x1929, 0x192B, 8, 0x1930, 0x1931, 8, + 0x1933, 0x1938, 8, 0x1A19, 0x1A1A, 8, 0x1A55, 0x1A55, 8, 0x1A57, 0x1A57, 8, + 0x1A61, 0x1A61, 8, 0x1A63, 0x1A64, 8, 0x1A6D, 0x1A72, 8, 0x1B04, 0x1B04, 8, + 0x1B35, 0x1B35, 8, 0x1B3B, 0x1B3B, 8, 0x1B3D, 0x1B41, 8, 0x1B43, 0x1B44, 8, + 0x1B82, 0x1B82, 8, 0x1BA1, 0x1BA1, 8, 0x1BA6, 0x1BA7, 8, 0x1BAA, 0x1BAA, 8, + 0x1BE7, 0x1BE7, 8, 0x1BEA, 0x1BEC, 8, 0x1BEE, 0x1BEE, 8, 0x1BF2, 0x1BF3, 8, + 0x1C24, 0x1C2B, 8, 0x1C34, 0x1C35, 8, 0x1CE1, 0x1CE1, 8, 0x1CF7, 0x1CF7, 8, + 0x302E, 0x302F, 8, 0xA823, 0xA824, 8, 0xA827, 0xA827, 8, 0xA880, 0xA881, 8, + 0xA8B4, 0xA8C3, 8, 0xA952, 0xA953, 8, 0xA983, 0xA983, 8, 0xA9B4, 0xA9B5, 8, + 0xA9BA, 0xA9BB, 8, 0xA9BE, 0xA9C0, 8, 0xAA2F, 0xAA30, 8, 0xAA33, 0xAA34, 8, + 0xAA4D, 0xAA4D, 8, 0xAA7B, 0xAA7B, 8, 0xAA7D, 0xAA7D, 8, 0xAAEB, 0xAAEB, 8, + 0xAAEE, 0xAAEF, 8, 0xAAF5, 0xAAF5, 8, 0xABE3, 0xABE4, 8, 0xABE6, 0xABE7, 8, + 0xABE9, 0xABEA, 8, 0xABEC, 0xABEC, 8, 0x11000, 0x11000, 8, 0x11002, 0x11002, 8, + 0x11082, 0x11082, 8, 0x110B0, 0x110B2, 8, 0x110B7, 0x110B8, 8, 0x1112C, 0x1112C, 8, + 0x11145, 0x11146, 8, 0x11182, 0x11182, 8, 0x111B3, 0x111B5, 8, 0x111BF, 0x111C0, 8, + 0x111CE, 0x111CE, 8, 0x1122C, 0x1122E, 8, 0x11232, 0x11233, 8, 0x11235, 0x11235, 8, + 0x112E0, 0x112E2, 8, 0x11302, 0x11303, 8, 0x1133E, 0x1133F, 8, 0x11341, 0x11344, 8, + 0x11347, 0x11348, 8, 0x1134B, 0x1134D, 8, 0x11357, 0x11357, 8, 0x11362, 0x11363, 8, + 0x113B8, 0x113BA, 8, 0x113C2, 0x113C2, 8, 0x113C5, 0x113C5, 8, 0x113C7, 0x113CA, 8, + 0x113CC, 0x113CD, 8, 0x113CF, 0x113CF, 8, 0x11435, 0x11437, 8, 0x11440, 0x11441, 8, + 0x11445, 0x11445, 8, 0x114B0, 0x114B2, 8, 0x114B9, 0x114B9, 8, 0x114BB, 0x114BE, 8, + 0x114C1, 0x114C1, 8, 0x115AF, 0x115B1, 8, 0x115B8, 0x115BB, 8, 0x115BE, 0x115BE, 8, + }; + } + + private static int[] rangeChunk8() { + return new int[] { + 0x11630, 0x11632, 8, 0x1163B, 0x1163C, 8, 0x1163E, 0x1163E, 8, 0x116AC, 0x116AC, 8, + 0x116AE, 0x116AF, 8, 0x116B6, 0x116B6, 8, 0x1171E, 0x1171E, 8, 0x11720, 0x11721, 8, + 0x11726, 0x11726, 8, 0x1182C, 0x1182E, 8, 0x11838, 0x11838, 8, 0x11930, 0x11935, 8, + 0x11937, 0x11938, 8, 0x1193D, 0x1193D, 8, 0x11940, 0x11940, 8, 0x11942, 0x11942, 8, + 0x119D1, 0x119D3, 8, 0x119DC, 0x119DF, 8, 0x119E4, 0x119E4, 8, 0x11A39, 0x11A39, 8, + 0x11A57, 0x11A58, 8, 0x11A97, 0x11A97, 8, 0x11B61, 0x11B61, 8, 0x11B65, 0x11B65, 8, + 0x11B67, 0x11B67, 8, 0x11C2F, 0x11C2F, 8, 0x11C3E, 0x11C3E, 8, 0x11CA9, 0x11CA9, 8, + 0x11CB1, 0x11CB1, 8, 0x11CB4, 0x11CB4, 8, 0x11D8A, 0x11D8E, 8, 0x11D93, 0x11D94, 8, + 0x11D96, 0x11D96, 8, 0x11EF5, 0x11EF6, 8, 0x11F03, 0x11F03, 8, 0x11F34, 0x11F35, 8, + 0x11F3E, 0x11F3F, 8, 0x11F41, 0x11F41, 8, 0x1612A, 0x1612C, 8, 0x16F51, 0x16F87, 8, + 0x16FF0, 0x16FF1, 8, 0x1D165, 0x1D166, 8, 0x1D16D, 0x1D172, 8, 0x30, 0x39, 9, + 0x660, 0x669, 9, 0x6F0, 0x6F9, 9, 0x7C0, 0x7C9, 9, 0x966, 0x96F, 9, + 0x9E6, 0x9EF, 9, 0xA66, 0xA6F, 9, 0xAE6, 0xAEF, 9, 0xB66, 0xB6F, 9, + 0xBE6, 0xBEF, 9, 0xC66, 0xC6F, 9, 0xCE6, 0xCEF, 9, 0xD66, 0xD6F, 9, + 0xDE6, 0xDEF, 9, 0xE50, 0xE59, 9, 0xED0, 0xED9, 9, 0xF20, 0xF29, 9, + 0x1040, 0x1049, 9, 0x1090, 0x1099, 9, 0x17E0, 0x17E9, 9, 0x1810, 0x1819, 9, + 0x1946, 0x194F, 9, 0x19D0, 0x19D9, 9, 0x1A80, 0x1A89, 9, 0x1A90, 0x1A99, 9, + 0x1B50, 0x1B59, 9, 0x1BB0, 0x1BB9, 9, 0x1C40, 0x1C49, 9, 0x1C50, 0x1C59, 9, + 0xA620, 0xA629, 9, 0xA8D0, 0xA8D9, 9, 0xA900, 0xA909, 9, 0xA9D0, 0xA9D9, 9, + 0xA9F0, 0xA9F9, 9, 0xAA50, 0xAA59, 9, 0xABF0, 0xABF9, 9, 0xFF10, 0xFF19, 9, + 0x104A0, 0x104A9, 9, 0x10D30, 0x10D39, 9, 0x10D40, 0x10D49, 9, 0x11066, 0x1106F, 9, + 0x110F0, 0x110F9, 9, 0x11136, 0x1113F, 9, 0x111D0, 0x111D9, 9, 0x112F0, 0x112F9, 9, + 0x11450, 0x11459, 9, 0x114D0, 0x114D9, 9, 0x11650, 0x11659, 9, 0x116C0, 0x116C9, 9, + 0x116D0, 0x116E3, 9, 0x11730, 0x11739, 9, 0x118E0, 0x118E9, 9, 0x11950, 0x11959, 9, + 0x11BF0, 0x11BF9, 9, 0x11C50, 0x11C59, 9, 0x11D50, 0x11D59, 9, 0x11DA0, 0x11DA9, 9, + 0x11DE0, 0x11DE9, 9, 0x11F50, 0x11F59, 9, 0x16130, 0x16139, 9, 0x16A60, 0x16A69, 9, + 0x16AC0, 0x16AC9, 9, 0x16B50, 0x16B59, 9, 0x16D70, 0x16D79, 9, 0x1CCF0, 0x1CCF9, 9, + 0x1D7CE, 0x1D7FF, 9, 0x1E140, 0x1E149, 9, 0x1E2F0, 0x1E2F9, 9, 0x1E4F0, 0x1E4F9, 9, + 0x1E5F1, 0x1E5FA, 9, 0x1E950, 0x1E959, 9, 0x1FBF0, 0x1FBF9, 9, 0x16EE, 0x16F0, 10, + 0x2160, 0x2182, 10, 0x2185, 0x2188, 10, 0x3007, 0x3007, 10, 0x3021, 0x3029, 10, + 0x3038, 0x303A, 10, 0xA6E6, 0xA6EF, 10, 0x10140, 0x10174, 10, 0x10341, 0x10341, 10, + 0x1034A, 0x1034A, 10, 0x103D1, 0x103D5, 10, 0x12400, 0x1246E, 10, 0x16FF4, 0x16FF6, 10, + 0xB2, 0xB3, 11, 0xB9, 0xB9, 11, 0xBC, 0xBE, 11, 0x9F4, 0x9F9, 11, + 0xB72, 0xB77, 11, 0xBF0, 0xBF2, 11, 0xC78, 0xC7E, 11, 0xD58, 0xD5E, 11, + 0xD70, 0xD78, 11, 0xF2A, 0xF33, 11, 0x1369, 0x137C, 11, 0x17F0, 0x17F9, 11, + 0x19DA, 0x19DA, 11, 0x2070, 0x2070, 11, 0x2074, 0x2079, 11, 0x2080, 0x2089, 11, + 0x2150, 0x215F, 11, 0x2189, 0x2189, 11, 0x2460, 0x249B, 11, 0x24EA, 0x24FF, 11, + 0x2776, 0x2793, 11, 0x2CFD, 0x2CFD, 11, 0x3192, 0x3195, 11, 0x3220, 0x3229, 11, + 0x3248, 0x324F, 11, 0x3251, 0x325F, 11, 0x3280, 0x3289, 11, 0x32B1, 0x32BF, 11, + 0xA830, 0xA835, 11, 0x10107, 0x10133, 11, 0x10175, 0x10178, 11, 0x1018A, 0x1018B, 11, + 0x102E1, 0x102FB, 11, 0x10320, 0x10323, 11, 0x10858, 0x1085F, 11, 0x10879, 0x1087F, 11, + 0x108A7, 0x108AF, 11, 0x108FB, 0x108FF, 11, 0x10916, 0x1091B, 11, 0x109BC, 0x109BD, 11, + 0x109C0, 0x109CF, 11, 0x109D2, 0x109FF, 11, 0x10A40, 0x10A48, 11, 0x10A7D, 0x10A7E, 11, + 0x10A9D, 0x10A9F, 11, 0x10AEB, 0x10AEF, 11, 0x10B58, 0x10B5F, 11, 0x10B78, 0x10B7F, 11, + 0x10BA9, 0x10BAF, 11, 0x10CFA, 0x10CFF, 11, 0x10E60, 0x10E7E, 11, 0x10F1D, 0x10F26, 11, + 0x10F51, 0x10F54, 11, 0x10FC5, 0x10FCB, 11, 0x11052, 0x11065, 11, 0x111E1, 0x111F4, 11, + 0x1173A, 0x1173B, 11, 0x118EA, 0x118F2, 11, 0x11C5A, 0x11C6C, 11, 0x11FC0, 0x11FD4, 11, + 0x16B5B, 0x16B61, 11, 0x16E80, 0x16E96, 11, 0x1D2C0, 0x1D2D3, 11, 0x1D2E0, 0x1D2F3, 11, + 0x1D360, 0x1D378, 11, 0x1E8C7, 0x1E8CF, 11, 0x1EC71, 0x1ECAB, 11, 0x1ECAD, 0x1ECAF, 11, + 0x1ECB1, 0x1ECB4, 11, 0x1ED01, 0x1ED2D, 11, 0x1ED2F, 0x1ED3D, 11, 0x1F100, 0x1F10C, 11, + 0x20, 0x20, 12, 0xA0, 0xA0, 12, 0x1680, 0x1680, 12, 0x2000, 0x200A, 12, + 0x202F, 0x202F, 12, 0x205F, 0x205F, 12, 0x3000, 0x3000, 12, 0x2028, 0x2028, 13, + 0x2029, 0x2029, 14, 0x0, 0x1F, 15, 0x7F, 0x9F, 15, 0xAD, 0xAD, 16, + 0x600, 0x605, 16, 0x61C, 0x61C, 16, 0x6DD, 0x6DD, 16, 0x70F, 0x70F, 16, + 0x890, 0x891, 16, 0x8E2, 0x8E2, 16, 0x180E, 0x180E, 16, 0x200B, 0x200F, 16, + 0x202A, 0x202E, 16, 0x2060, 0x2064, 16, 0x2066, 0x206F, 16, 0xFEFF, 0xFEFF, 16, + 0xFFF9, 0xFFFB, 16, 0x110BD, 0x110BD, 16, 0x110CD, 0x110CD, 16, 0x13430, 0x1343F, 16, + 0x1BCA0, 0x1BCA3, 16, 0x1D173, 0x1D17A, 16, 0xE0001, 0xE0001, 16, 0xE0020, 0xE007F, 16, + 0xE000, 0xF8FF, 17, 0xF0000, 0xFFFFD, 17, 0x100000, 0x10FFFD, 17, 0xD800, 0xDFFF, 18, + 0x2D, 0x2D, 19, 0x58A, 0x58A, 19, 0x5BE, 0x5BE, 19, 0x1400, 0x1400, 19, + 0x1806, 0x1806, 19, 0x2010, 0x2015, 19, 0x2E17, 0x2E17, 19, 0x2E1A, 0x2E1A, 19, + 0x2E3A, 0x2E3B, 19, 0x2E40, 0x2E40, 19, 0x2E5D, 0x2E5D, 19, 0x301C, 0x301C, 19, + 0x3030, 0x3030, 19, 0x30A0, 0x30A0, 19, 0xFE31, 0xFE32, 19, 0xFE58, 0xFE58, 19, + 0xFE63, 0xFE63, 19, 0xFF0D, 0xFF0D, 19, 0x10D6E, 0x10D6E, 19, 0x10EAD, 0x10EAD, 19, + 0x28, 0x28, 20, 0x5B, 0x5B, 20, 0x7B, 0x7B, 20, 0xF3A, 0xF3A, 20, + 0xF3C, 0xF3C, 20, 0x169B, 0x169B, 20, 0x201A, 0x201A, 20, 0x201E, 0x201E, 20, + 0x2045, 0x2045, 20, 0x207D, 0x207D, 20, 0x208D, 0x208D, 20, 0x2308, 0x2308, 20, + 0x230A, 0x230A, 20, 0x2329, 0x2329, 20, 0x2768, 0x2768, 20, 0x276A, 0x276A, 20, + 0x276C, 0x276C, 20, 0x276E, 0x276E, 20, 0x2770, 0x2770, 20, 0x2772, 0x2772, 20, + 0x2774, 0x2774, 20, 0x27C5, 0x27C5, 20, 0x27E6, 0x27E6, 20, 0x27E8, 0x27E8, 20, + 0x27EA, 0x27EA, 20, 0x27EC, 0x27EC, 20, 0x27EE, 0x27EE, 20, 0x2983, 0x2983, 20, + 0x2985, 0x2985, 20, 0x2987, 0x2987, 20, 0x2989, 0x2989, 20, 0x298B, 0x298B, 20, + 0x298D, 0x298D, 20, 0x298F, 0x298F, 20, 0x2991, 0x2991, 20, 0x2993, 0x2993, 20, + 0x2995, 0x2995, 20, 0x2997, 0x2997, 20, 0x29D8, 0x29D8, 20, 0x29DA, 0x29DA, 20, + 0x29FC, 0x29FC, 20, 0x2E22, 0x2E22, 20, 0x2E24, 0x2E24, 20, 0x2E26, 0x2E26, 20, + 0x2E28, 0x2E28, 20, 0x2E42, 0x2E42, 20, 0x2E55, 0x2E55, 20, 0x2E57, 0x2E57, 20, + 0x2E59, 0x2E59, 20, 0x2E5B, 0x2E5B, 20, 0x3008, 0x3008, 20, 0x300A, 0x300A, 20, + 0x300C, 0x300C, 20, 0x300E, 0x300E, 20, 0x3010, 0x3010, 20, 0x3014, 0x3014, 20, + 0x3016, 0x3016, 20, 0x3018, 0x3018, 20, 0x301A, 0x301A, 20, 0x301D, 0x301D, 20, + 0xFD3F, 0xFD3F, 20, 0xFE17, 0xFE17, 20, 0xFE35, 0xFE35, 20, 0xFE37, 0xFE37, 20, + 0xFE39, 0xFE39, 20, 0xFE3B, 0xFE3B, 20, 0xFE3D, 0xFE3D, 20, 0xFE3F, 0xFE3F, 20, + 0xFE41, 0xFE41, 20, 0xFE43, 0xFE43, 20, 0xFE47, 0xFE47, 20, 0xFE59, 0xFE59, 20, + 0xFE5B, 0xFE5B, 20, 0xFE5D, 0xFE5D, 20, 0xFF08, 0xFF08, 20, 0xFF3B, 0xFF3B, 20, + 0xFF5B, 0xFF5B, 20, 0xFF5F, 0xFF5F, 20, 0xFF62, 0xFF62, 20, 0x29, 0x29, 21, + 0x5D, 0x5D, 21, 0x7D, 0x7D, 21, 0xF3B, 0xF3B, 21, 0xF3D, 0xF3D, 21, + 0x169C, 0x169C, 21, 0x2046, 0x2046, 21, 0x207E, 0x207E, 21, 0x208E, 0x208E, 21, + 0x2309, 0x2309, 21, 0x230B, 0x230B, 21, 0x232A, 0x232A, 21, 0x2769, 0x2769, 21, + 0x276B, 0x276B, 21, 0x276D, 0x276D, 21, 0x276F, 0x276F, 21, 0x2771, 0x2771, 21, + 0x2773, 0x2773, 21, 0x2775, 0x2775, 21, 0x27C6, 0x27C6, 21, 0x27E7, 0x27E7, 21, + 0x27E9, 0x27E9, 21, 0x27EB, 0x27EB, 21, 0x27ED, 0x27ED, 21, 0x27EF, 0x27EF, 21, + 0x2984, 0x2984, 21, 0x2986, 0x2986, 21, 0x2988, 0x2988, 21, 0x298A, 0x298A, 21, + 0x298C, 0x298C, 21, 0x298E, 0x298E, 21, 0x2990, 0x2990, 21, 0x2992, 0x2992, 21, + 0x2994, 0x2994, 21, 0x2996, 0x2996, 21, 0x2998, 0x2998, 21, 0x29D9, 0x29D9, 21, + 0x29DB, 0x29DB, 21, 0x29FD, 0x29FD, 21, 0x2E23, 0x2E23, 21, 0x2E25, 0x2E25, 21, + 0x2E27, 0x2E27, 21, 0x2E29, 0x2E29, 21, 0x2E56, 0x2E56, 21, 0x2E58, 0x2E58, 21, + 0x2E5A, 0x2E5A, 21, 0x2E5C, 0x2E5C, 21, 0x3009, 0x3009, 21, 0x300B, 0x300B, 21, + 0x300D, 0x300D, 21, 0x300F, 0x300F, 21, 0x3011, 0x3011, 21, 0x3015, 0x3015, 21, + 0x3017, 0x3017, 21, 0x3019, 0x3019, 21, 0x301B, 0x301B, 21, 0x301E, 0x301F, 21, + 0xFD3E, 0xFD3E, 21, 0xFE18, 0xFE18, 21, 0xFE36, 0xFE36, 21, 0xFE38, 0xFE38, 21, + 0xFE3A, 0xFE3A, 21, 0xFE3C, 0xFE3C, 21, 0xFE3E, 0xFE3E, 21, 0xFE40, 0xFE40, 21, + }; + } + + private static int[] rangeChunk9() { + return new int[] { + 0xFE42, 0xFE42, 21, 0xFE44, 0xFE44, 21, 0xFE48, 0xFE48, 21, 0xFE5A, 0xFE5A, 21, + 0xFE5C, 0xFE5C, 21, 0xFE5E, 0xFE5E, 21, 0xFF09, 0xFF09, 21, 0xFF3D, 0xFF3D, 21, + 0xFF5D, 0xFF5D, 21, 0xFF60, 0xFF60, 21, 0xFF63, 0xFF63, 21, 0x5F, 0x5F, 22, + 0x203F, 0x2040, 22, 0x2054, 0x2054, 22, 0xFE33, 0xFE34, 22, 0xFE4D, 0xFE4F, 22, + 0xFF3F, 0xFF3F, 22, 0x21, 0x23, 23, 0x25, 0x27, 23, 0x2A, 0x2A, 23, + 0x2C, 0x2C, 23, 0x2E, 0x2F, 23, 0x3A, 0x3B, 23, 0x3F, 0x40, 23, + 0x5C, 0x5C, 23, 0xA1, 0xA1, 23, 0xA7, 0xA7, 23, 0xB6, 0xB7, 23, + 0xBF, 0xBF, 23, 0x37E, 0x37E, 23, 0x387, 0x387, 23, 0x55A, 0x55F, 23, + 0x589, 0x589, 23, 0x5C0, 0x5C0, 23, 0x5C3, 0x5C3, 23, 0x5C6, 0x5C6, 23, + 0x5F3, 0x5F4, 23, 0x609, 0x60A, 23, 0x60C, 0x60D, 23, 0x61B, 0x61B, 23, + 0x61D, 0x61F, 23, 0x66A, 0x66D, 23, 0x6D4, 0x6D4, 23, 0x700, 0x70D, 23, + 0x7F7, 0x7F9, 23, 0x830, 0x83E, 23, 0x85E, 0x85E, 23, 0x964, 0x965, 23, + 0x970, 0x970, 23, 0x9FD, 0x9FD, 23, 0xA76, 0xA76, 23, 0xAF0, 0xAF0, 23, + 0xC77, 0xC77, 23, 0xC84, 0xC84, 23, 0xDF4, 0xDF4, 23, 0xE4F, 0xE4F, 23, + 0xE5A, 0xE5B, 23, 0xF04, 0xF12, 23, 0xF14, 0xF14, 23, 0xF85, 0xF85, 23, + 0xFD0, 0xFD4, 23, 0xFD9, 0xFDA, 23, 0x104A, 0x104F, 23, 0x10FB, 0x10FB, 23, + 0x1360, 0x1368, 23, 0x166E, 0x166E, 23, 0x16EB, 0x16ED, 23, 0x1735, 0x1736, 23, + 0x17D4, 0x17D6, 23, 0x17D8, 0x17DA, 23, 0x1800, 0x1805, 23, 0x1807, 0x180A, 23, + 0x1944, 0x1945, 23, 0x1A1E, 0x1A1F, 23, 0x1AA0, 0x1AA6, 23, 0x1AA8, 0x1AAD, 23, + 0x1B4E, 0x1B4F, 23, 0x1B5A, 0x1B60, 23, 0x1B7D, 0x1B7F, 23, 0x1BFC, 0x1BFF, 23, + 0x1C3B, 0x1C3F, 23, 0x1C7E, 0x1C7F, 23, 0x1CC0, 0x1CC7, 23, 0x1CD3, 0x1CD3, 23, + 0x2016, 0x2017, 23, 0x2020, 0x2027, 23, 0x2030, 0x2038, 23, 0x203B, 0x203E, 23, + 0x2041, 0x2043, 23, 0x2047, 0x2051, 23, 0x2053, 0x2053, 23, 0x2055, 0x205E, 23, + 0x2CF9, 0x2CFC, 23, 0x2CFE, 0x2CFF, 23, 0x2D70, 0x2D70, 23, 0x2E00, 0x2E01, 23, + 0x2E06, 0x2E08, 23, 0x2E0B, 0x2E0B, 23, 0x2E0E, 0x2E16, 23, 0x2E18, 0x2E19, 23, + 0x2E1B, 0x2E1B, 23, 0x2E1E, 0x2E1F, 23, 0x2E2A, 0x2E2E, 23, 0x2E30, 0x2E39, 23, + 0x2E3C, 0x2E3F, 23, 0x2E41, 0x2E41, 23, 0x2E43, 0x2E4F, 23, 0x2E52, 0x2E54, 23, + 0x3001, 0x3003, 23, 0x303D, 0x303D, 23, 0x30FB, 0x30FB, 23, 0xA4FE, 0xA4FF, 23, + 0xA60D, 0xA60F, 23, 0xA673, 0xA673, 23, 0xA67E, 0xA67E, 23, 0xA6F2, 0xA6F7, 23, + 0xA874, 0xA877, 23, 0xA8CE, 0xA8CF, 23, 0xA8F8, 0xA8FA, 23, 0xA8FC, 0xA8FC, 23, + 0xA92E, 0xA92F, 23, 0xA95F, 0xA95F, 23, 0xA9C1, 0xA9CD, 23, 0xA9DE, 0xA9DF, 23, + 0xAA5C, 0xAA5F, 23, 0xAADE, 0xAADF, 23, 0xAAF0, 0xAAF1, 23, 0xABEB, 0xABEB, 23, + 0xFE10, 0xFE16, 23, 0xFE19, 0xFE19, 23, 0xFE30, 0xFE30, 23, 0xFE45, 0xFE46, 23, + 0xFE49, 0xFE4C, 23, 0xFE50, 0xFE52, 23, 0xFE54, 0xFE57, 23, 0xFE5F, 0xFE61, 23, + 0xFE68, 0xFE68, 23, 0xFE6A, 0xFE6B, 23, 0xFF01, 0xFF03, 23, 0xFF05, 0xFF07, 23, + 0xFF0A, 0xFF0A, 23, 0xFF0C, 0xFF0C, 23, 0xFF0E, 0xFF0F, 23, 0xFF1A, 0xFF1B, 23, + 0xFF1F, 0xFF20, 23, 0xFF3C, 0xFF3C, 23, 0xFF61, 0xFF61, 23, 0xFF64, 0xFF65, 23, + 0x10100, 0x10102, 23, 0x1039F, 0x1039F, 23, 0x103D0, 0x103D0, 23, 0x1056F, 0x1056F, 23, + 0x10857, 0x10857, 23, 0x1091F, 0x1091F, 23, 0x1093F, 0x1093F, 23, 0x10A50, 0x10A58, 23, + 0x10A7F, 0x10A7F, 23, 0x10AF0, 0x10AF6, 23, 0x10B39, 0x10B3F, 23, 0x10B99, 0x10B9C, 23, + 0x10ED0, 0x10ED0, 23, 0x10F55, 0x10F59, 23, 0x10F86, 0x10F89, 23, 0x11047, 0x1104D, 23, + 0x110BB, 0x110BC, 23, 0x110BE, 0x110C1, 23, 0x11140, 0x11143, 23, 0x11174, 0x11175, 23, + 0x111C5, 0x111C8, 23, 0x111CD, 0x111CD, 23, 0x111DB, 0x111DB, 23, 0x111DD, 0x111DF, 23, + 0x11238, 0x1123D, 23, 0x112A9, 0x112A9, 23, 0x113D4, 0x113D5, 23, 0x113D7, 0x113D8, 23, + 0x1144B, 0x1144F, 23, 0x1145A, 0x1145B, 23, 0x1145D, 0x1145D, 23, 0x114C6, 0x114C6, 23, + 0x115C1, 0x115D7, 23, 0x11641, 0x11643, 23, 0x11660, 0x1166C, 23, 0x116B9, 0x116B9, 23, + 0x1173C, 0x1173E, 23, 0x1183B, 0x1183B, 23, 0x11944, 0x11946, 23, 0x119E2, 0x119E2, 23, + 0x11A3F, 0x11A46, 23, 0x11A9A, 0x11A9C, 23, 0x11A9E, 0x11AA2, 23, 0x11B00, 0x11B09, 23, + 0x11BE1, 0x11BE1, 23, 0x11C41, 0x11C45, 23, 0x11C70, 0x11C71, 23, 0x11EF7, 0x11EF8, 23, + 0x11F43, 0x11F4F, 23, 0x11FFF, 0x11FFF, 23, 0x12470, 0x12474, 23, 0x12FF1, 0x12FF2, 23, + 0x16A6E, 0x16A6F, 23, 0x16AF5, 0x16AF5, 23, 0x16B37, 0x16B3B, 23, 0x16B44, 0x16B44, 23, + 0x16D6D, 0x16D6F, 23, 0x16E97, 0x16E9A, 23, 0x16FE2, 0x16FE2, 23, 0x1BC9F, 0x1BC9F, 23, + 0x1DA87, 0x1DA8B, 23, 0x1E5FF, 0x1E5FF, 23, 0x1E95E, 0x1E95F, 23, 0x2B, 0x2B, 24, + 0x3C, 0x3E, 24, 0x7C, 0x7C, 24, 0x7E, 0x7E, 24, 0xAC, 0xAC, 24, + 0xB1, 0xB1, 24, 0xD7, 0xD7, 24, 0xF7, 0xF7, 24, 0x3F6, 0x3F6, 24, + 0x606, 0x608, 24, 0x2044, 0x2044, 24, 0x2052, 0x2052, 24, 0x207A, 0x207C, 24, + 0x208A, 0x208C, 24, 0x2118, 0x2118, 24, 0x2140, 0x2144, 24, 0x214B, 0x214B, 24, + 0x2190, 0x2194, 24, 0x219A, 0x219B, 24, 0x21A0, 0x21A0, 24, 0x21A3, 0x21A3, 24, + 0x21A6, 0x21A6, 24, 0x21AE, 0x21AE, 24, 0x21CE, 0x21CF, 24, 0x21D2, 0x21D2, 24, + 0x21D4, 0x21D4, 24, 0x21F4, 0x22FF, 24, 0x2320, 0x2321, 24, 0x237C, 0x237C, 24, + 0x239B, 0x23B3, 24, 0x23DC, 0x23E1, 24, 0x25B7, 0x25B7, 24, 0x25C1, 0x25C1, 24, + 0x25F8, 0x25FF, 24, 0x266F, 0x266F, 24, 0x27C0, 0x27C4, 24, 0x27C7, 0x27E5, 24, + 0x27F0, 0x27FF, 24, 0x2900, 0x2982, 24, 0x2999, 0x29D7, 24, 0x29DC, 0x29FB, 24, + 0x29FE, 0x2AFF, 24, 0x2B30, 0x2B44, 24, 0x2B47, 0x2B4C, 24, 0xFB29, 0xFB29, 24, + 0xFE62, 0xFE62, 24, 0xFE64, 0xFE66, 24, 0xFF0B, 0xFF0B, 24, 0xFF1C, 0xFF1E, 24, + 0xFF5C, 0xFF5C, 24, 0xFF5E, 0xFF5E, 24, 0xFFE2, 0xFFE2, 24, 0xFFE9, 0xFFEC, 24, + 0x10D8E, 0x10D8F, 24, 0x1CEF0, 0x1CEF0, 24, 0x1D6C1, 0x1D6C1, 24, 0x1D6DB, 0x1D6DB, 24, + 0x1D6FB, 0x1D6FB, 24, 0x1D715, 0x1D715, 24, 0x1D735, 0x1D735, 24, 0x1D74F, 0x1D74F, 24, + 0x1D76F, 0x1D76F, 24, 0x1D789, 0x1D789, 24, 0x1D7A9, 0x1D7A9, 24, 0x1D7C3, 0x1D7C3, 24, + 0x1EEF0, 0x1EEF1, 24, 0x1F8D0, 0x1F8D8, 24, 0x24, 0x24, 25, 0xA2, 0xA5, 25, + 0x58F, 0x58F, 25, 0x60B, 0x60B, 25, 0x7FE, 0x7FF, 25, 0x9F2, 0x9F3, 25, + 0x9FB, 0x9FB, 25, 0xAF1, 0xAF1, 25, 0xBF9, 0xBF9, 25, 0xE3F, 0xE3F, 25, + 0x17DB, 0x17DB, 25, 0x20A0, 0x20C1, 25, 0xA838, 0xA838, 25, 0xFDFC, 0xFDFC, 25, + 0xFE69, 0xFE69, 25, 0xFF04, 0xFF04, 25, 0xFFE0, 0xFFE1, 25, 0xFFE5, 0xFFE6, 25, + 0x11FDD, 0x11FE0, 25, 0x1E2FF, 0x1E2FF, 25, 0x1ECB0, 0x1ECB0, 25, 0x5E, 0x5E, 26, + 0x60, 0x60, 26, 0xA8, 0xA8, 26, 0xAF, 0xAF, 26, 0xB4, 0xB4, 26, + 0xB8, 0xB8, 26, 0x2C2, 0x2C5, 26, 0x2D2, 0x2DF, 26, 0x2E5, 0x2EB, 26, + 0x2ED, 0x2ED, 26, 0x2EF, 0x2FF, 26, 0x375, 0x375, 26, 0x384, 0x385, 26, + 0x888, 0x888, 26, 0x1FBD, 0x1FBD, 26, 0x1FBF, 0x1FC1, 26, 0x1FCD, 0x1FCF, 26, + 0x1FDD, 0x1FDF, 26, 0x1FED, 0x1FEF, 26, 0x1FFD, 0x1FFE, 26, 0x309B, 0x309C, 26, + 0xA700, 0xA716, 26, 0xA720, 0xA721, 26, 0xA789, 0xA78A, 26, 0xAB5B, 0xAB5B, 26, + 0xAB6A, 0xAB6B, 26, 0xFBB2, 0xFBC2, 26, 0xFF3E, 0xFF3E, 26, 0xFF40, 0xFF40, 26, + 0xFFE3, 0xFFE3, 26, 0x1F3FB, 0x1F3FF, 26, 0xA6, 0xA6, 27, 0xA9, 0xA9, 27, + 0xAE, 0xAE, 27, 0xB0, 0xB0, 27, 0x482, 0x482, 27, 0x58D, 0x58E, 27, + 0x60E, 0x60F, 27, 0x6DE, 0x6DE, 27, 0x6E9, 0x6E9, 27, 0x6FD, 0x6FE, 27, + 0x7F6, 0x7F6, 27, 0x9FA, 0x9FA, 27, 0xB70, 0xB70, 27, 0xBF3, 0xBF8, 27, + 0xBFA, 0xBFA, 27, 0xC7F, 0xC7F, 27, 0xD4F, 0xD4F, 27, 0xD79, 0xD79, 27, + 0xF01, 0xF03, 27, 0xF13, 0xF13, 27, 0xF15, 0xF17, 27, 0xF1A, 0xF1F, 27, + 0xF34, 0xF34, 27, 0xF36, 0xF36, 27, 0xF38, 0xF38, 27, 0xFBE, 0xFC5, 27, + 0xFC7, 0xFCC, 27, 0xFCE, 0xFCF, 27, 0xFD5, 0xFD8, 27, 0x109E, 0x109F, 27, + 0x1390, 0x1399, 27, 0x166D, 0x166D, 27, 0x1940, 0x1940, 27, 0x19DE, 0x19FF, 27, + 0x1B61, 0x1B6A, 27, 0x1B74, 0x1B7C, 27, 0x2100, 0x2101, 27, 0x2103, 0x2106, 27, + 0x2108, 0x2109, 27, 0x2114, 0x2114, 27, 0x2116, 0x2117, 27, 0x211E, 0x2123, 27, + 0x2125, 0x2125, 27, 0x2127, 0x2127, 27, 0x2129, 0x2129, 27, 0x212E, 0x212E, 27, + 0x213A, 0x213B, 27, 0x214A, 0x214A, 27, 0x214C, 0x214D, 27, 0x214F, 0x214F, 27, + 0x218A, 0x218B, 27, 0x2195, 0x2199, 27, 0x219C, 0x219F, 27, 0x21A1, 0x21A2, 27, + 0x21A4, 0x21A5, 27, 0x21A7, 0x21AD, 27, 0x21AF, 0x21CD, 27, 0x21D0, 0x21D1, 27, + 0x21D3, 0x21D3, 27, 0x21D5, 0x21F3, 27, 0x2300, 0x2307, 27, 0x230C, 0x231F, 27, + 0x2322, 0x2328, 27, 0x232B, 0x237B, 27, 0x237D, 0x239A, 27, 0x23B4, 0x23DB, 27, + 0x23E2, 0x2429, 27, 0x2440, 0x244A, 27, 0x249C, 0x24E9, 27, 0x2500, 0x25B6, 27, + }; + } + + private static int[] rangeChunk10() { + return new int[] { + 0x25B8, 0x25C0, 27, 0x25C2, 0x25F7, 27, 0x2600, 0x266E, 27, 0x2670, 0x2767, 27, + 0x2794, 0x27BF, 27, 0x2800, 0x28FF, 27, 0x2B00, 0x2B2F, 27, 0x2B45, 0x2B46, 27, + 0x2B4D, 0x2B73, 27, 0x2B76, 0x2BFF, 27, 0x2CE5, 0x2CEA, 27, 0x2E50, 0x2E51, 27, + 0x2E80, 0x2E99, 27, 0x2E9B, 0x2EF3, 27, 0x2F00, 0x2FD5, 27, 0x2FF0, 0x2FFF, 27, + 0x3004, 0x3004, 27, 0x3012, 0x3013, 27, 0x3020, 0x3020, 27, 0x3036, 0x3037, 27, + 0x303E, 0x303F, 27, 0x3190, 0x3191, 27, 0x3196, 0x319F, 27, 0x31C0, 0x31E5, 27, + 0x31EF, 0x31EF, 27, 0x3200, 0x321E, 27, 0x322A, 0x3247, 27, 0x3250, 0x3250, 27, + 0x3260, 0x327F, 27, 0x328A, 0x32B0, 27, 0x32C0, 0x33FF, 27, 0x4DC0, 0x4DFF, 27, + 0xA490, 0xA4C6, 27, 0xA828, 0xA82B, 27, 0xA836, 0xA837, 27, 0xA839, 0xA839, 27, + 0xAA77, 0xAA79, 27, 0xFBC3, 0xFBD2, 27, 0xFD40, 0xFD4F, 27, 0xFD90, 0xFD91, 27, + 0xFDC8, 0xFDCF, 27, 0xFDFD, 0xFDFF, 27, 0xFFE4, 0xFFE4, 27, 0xFFE8, 0xFFE8, 27, + 0xFFED, 0xFFEE, 27, 0xFFFC, 0xFFFD, 27, 0x10137, 0x1013F, 27, 0x10179, 0x10189, 27, + 0x1018C, 0x1018E, 27, 0x10190, 0x1019C, 27, 0x101A0, 0x101A0, 27, 0x101D0, 0x101FC, 27, + 0x10877, 0x10878, 27, 0x10AC8, 0x10AC8, 27, 0x10ED1, 0x10ED8, 27, 0x1173F, 0x1173F, 27, + 0x11FD5, 0x11FDC, 27, 0x11FE1, 0x11FF1, 27, 0x16B3C, 0x16B3F, 27, 0x16B45, 0x16B45, 27, + 0x1BC9C, 0x1BC9C, 27, 0x1CC00, 0x1CCEF, 27, 0x1CCFA, 0x1CCFC, 27, 0x1CD00, 0x1CEB3, 27, + 0x1CEBA, 0x1CED0, 27, 0x1CEE0, 0x1CEEF, 27, 0x1CF50, 0x1CFC3, 27, 0x1D000, 0x1D0F5, 27, + 0x1D100, 0x1D126, 27, 0x1D129, 0x1D164, 27, 0x1D16A, 0x1D16C, 27, 0x1D183, 0x1D184, 27, + 0x1D18C, 0x1D1A9, 27, 0x1D1AE, 0x1D1EA, 27, 0x1D200, 0x1D241, 27, 0x1D245, 0x1D245, 27, + 0x1D300, 0x1D356, 27, 0x1D800, 0x1D9FF, 27, 0x1DA37, 0x1DA3A, 27, 0x1DA6D, 0x1DA74, 27, + 0x1DA76, 0x1DA83, 27, 0x1DA85, 0x1DA86, 27, 0x1E14F, 0x1E14F, 27, 0x1ECAC, 0x1ECAC, 27, + 0x1ED2E, 0x1ED2E, 27, 0x1F000, 0x1F02B, 27, 0x1F030, 0x1F093, 27, 0x1F0A0, 0x1F0AE, 27, + 0x1F0B1, 0x1F0BF, 27, 0x1F0C1, 0x1F0CF, 27, 0x1F0D1, 0x1F0F5, 27, 0x1F10D, 0x1F1AD, 27, + 0x1F1E6, 0x1F202, 27, 0x1F210, 0x1F23B, 27, 0x1F240, 0x1F248, 27, 0x1F250, 0x1F251, 27, + 0x1F260, 0x1F265, 27, 0x1F300, 0x1F3FA, 27, 0x1F400, 0x1F6D8, 27, 0x1F6DC, 0x1F6EC, 27, + 0x1F6F0, 0x1F6FC, 27, 0x1F700, 0x1F7D9, 27, 0x1F7E0, 0x1F7EB, 27, 0x1F7F0, 0x1F7F0, 27, + 0x1F800, 0x1F80B, 27, 0x1F810, 0x1F847, 27, 0x1F850, 0x1F859, 27, 0x1F860, 0x1F887, 27, + 0x1F890, 0x1F8AD, 27, 0x1F8B0, 0x1F8BB, 27, 0x1F8C0, 0x1F8C1, 27, 0x1F900, 0x1FA57, 27, + 0x1FA60, 0x1FA6D, 27, 0x1FA70, 0x1FA7C, 27, 0x1FA80, 0x1FA8A, 27, 0x1FA8E, 0x1FAC6, 27, + 0x1FAC8, 0x1FAC8, 27, 0x1FACD, 0x1FADC, 27, 0x1FADF, 0x1FAEA, 27, 0x1FAEF, 0x1FAF8, 27, + 0x1FB00, 0x1FB92, 27, 0x1FB94, 0x1FBEF, 27, 0x1FBFA, 0x1FBFA, 27, 0xAB, 0xAB, 28, + 0x2018, 0x2018, 28, 0x201B, 0x201C, 28, 0x201F, 0x201F, 28, 0x2039, 0x2039, 28, + 0x2E02, 0x2E02, 28, 0x2E04, 0x2E04, 28, 0x2E09, 0x2E09, 28, 0x2E0C, 0x2E0C, 28, + 0x2E1C, 0x2E1C, 28, 0x2E20, 0x2E20, 28, 0xBB, 0xBB, 29, 0x2019, 0x2019, 29, + 0x201D, 0x201D, 29, 0x203A, 0x203A, 29, 0x2E03, 0x2E03, 29, 0x2E05, 0x2E05, 29, + 0x2E0A, 0x2E0A, 29, 0x2E0D, 0x2E0D, 29, 0x2E1D, 0x2E1D, 29, 0x2E21, 0x2E21, 29, + 0x378, 0x379, 30, 0x380, 0x383, 30, 0x38B, 0x38B, 30, 0x38D, 0x38D, 30, + 0x3A2, 0x3A2, 30, 0x530, 0x530, 30, 0x557, 0x558, 30, 0x58B, 0x58C, 30, + 0x590, 0x590, 30, 0x5C8, 0x5CF, 30, 0x5EB, 0x5EE, 30, 0x5F5, 0x5FF, 30, + 0x70E, 0x70E, 30, 0x74B, 0x74C, 30, 0x7B2, 0x7BF, 30, 0x7FB, 0x7FC, 30, + 0x82E, 0x82F, 30, 0x83F, 0x83F, 30, 0x85C, 0x85D, 30, 0x85F, 0x85F, 30, + 0x86B, 0x86F, 30, 0x892, 0x896, 30, 0x984, 0x984, 30, 0x98D, 0x98E, 30, + 0x991, 0x992, 30, 0x9A9, 0x9A9, 30, 0x9B1, 0x9B1, 30, 0x9B3, 0x9B5, 30, + 0x9BA, 0x9BB, 30, 0x9C5, 0x9C6, 30, 0x9C9, 0x9CA, 30, 0x9CF, 0x9D6, 30, + 0x9D8, 0x9DB, 30, 0x9DE, 0x9DE, 30, 0x9E4, 0x9E5, 30, 0x9FF, 0xA00, 30, + 0xA04, 0xA04, 30, 0xA0B, 0xA0E, 30, 0xA11, 0xA12, 30, 0xA29, 0xA29, 30, + 0xA31, 0xA31, 30, 0xA34, 0xA34, 30, 0xA37, 0xA37, 30, 0xA3A, 0xA3B, 30, + 0xA3D, 0xA3D, 30, 0xA43, 0xA46, 30, 0xA49, 0xA4A, 30, 0xA4E, 0xA50, 30, + 0xA52, 0xA58, 30, 0xA5D, 0xA5D, 30, 0xA5F, 0xA65, 30, 0xA77, 0xA80, 30, + 0xA84, 0xA84, 30, 0xA8E, 0xA8E, 30, 0xA92, 0xA92, 30, 0xAA9, 0xAA9, 30, + 0xAB1, 0xAB1, 30, 0xAB4, 0xAB4, 30, 0xABA, 0xABB, 30, 0xAC6, 0xAC6, 30, + 0xACA, 0xACA, 30, 0xACE, 0xACF, 30, 0xAD1, 0xADF, 30, 0xAE4, 0xAE5, 30, + 0xAF2, 0xAF8, 30, 0xB00, 0xB00, 30, 0xB04, 0xB04, 30, 0xB0D, 0xB0E, 30, + 0xB11, 0xB12, 30, 0xB29, 0xB29, 30, 0xB31, 0xB31, 30, 0xB34, 0xB34, 30, + 0xB3A, 0xB3B, 30, 0xB45, 0xB46, 30, 0xB49, 0xB4A, 30, 0xB4E, 0xB54, 30, + 0xB58, 0xB5B, 30, 0xB5E, 0xB5E, 30, 0xB64, 0xB65, 30, 0xB78, 0xB81, 30, + 0xB84, 0xB84, 30, 0xB8B, 0xB8D, 30, 0xB91, 0xB91, 30, 0xB96, 0xB98, 30, + 0xB9B, 0xB9B, 30, 0xB9D, 0xB9D, 30, 0xBA0, 0xBA2, 30, 0xBA5, 0xBA7, 30, + 0xBAB, 0xBAD, 30, 0xBBA, 0xBBD, 30, 0xBC3, 0xBC5, 30, 0xBC9, 0xBC9, 30, + 0xBCE, 0xBCF, 30, 0xBD1, 0xBD6, 30, 0xBD8, 0xBE5, 30, 0xBFB, 0xBFF, 30, + 0xC0D, 0xC0D, 30, 0xC11, 0xC11, 30, 0xC29, 0xC29, 30, 0xC3A, 0xC3B, 30, + 0xC45, 0xC45, 30, 0xC49, 0xC49, 30, 0xC4E, 0xC54, 30, 0xC57, 0xC57, 30, + 0xC5B, 0xC5B, 30, 0xC5E, 0xC5F, 30, 0xC64, 0xC65, 30, 0xC70, 0xC76, 30, + 0xC8D, 0xC8D, 30, 0xC91, 0xC91, 30, 0xCA9, 0xCA9, 30, 0xCB4, 0xCB4, 30, + 0xCBA, 0xCBB, 30, 0xCC5, 0xCC5, 30, 0xCC9, 0xCC9, 30, 0xCCE, 0xCD4, 30, + 0xCD7, 0xCDB, 30, 0xCDF, 0xCDF, 30, 0xCE4, 0xCE5, 30, 0xCF0, 0xCF0, 30, + 0xCF4, 0xCFF, 30, 0xD0D, 0xD0D, 30, 0xD11, 0xD11, 30, 0xD45, 0xD45, 30, + 0xD49, 0xD49, 30, 0xD50, 0xD53, 30, 0xD64, 0xD65, 30, 0xD80, 0xD80, 30, + 0xD84, 0xD84, 30, 0xD97, 0xD99, 30, 0xDB2, 0xDB2, 30, 0xDBC, 0xDBC, 30, + 0xDBE, 0xDBF, 30, 0xDC7, 0xDC9, 30, 0xDCB, 0xDCE, 30, 0xDD5, 0xDD5, 30, + 0xDD7, 0xDD7, 30, 0xDE0, 0xDE5, 30, 0xDF0, 0xDF1, 30, 0xDF5, 0xE00, 30, + 0xE3B, 0xE3E, 30, 0xE5C, 0xE80, 30, 0xE83, 0xE83, 30, 0xE85, 0xE85, 30, + 0xE8B, 0xE8B, 30, 0xEA4, 0xEA4, 30, 0xEA6, 0xEA6, 30, 0xEBE, 0xEBF, 30, + 0xEC5, 0xEC5, 30, 0xEC7, 0xEC7, 30, 0xECF, 0xECF, 30, 0xEDA, 0xEDB, 30, + 0xEE0, 0xEFF, 30, 0xF48, 0xF48, 30, 0xF6D, 0xF70, 30, 0xF98, 0xF98, 30, + 0xFBD, 0xFBD, 30, 0xFCD, 0xFCD, 30, 0xFDB, 0xFFF, 30, 0x10C6, 0x10C6, 30, + 0x10C8, 0x10CC, 30, 0x10CE, 0x10CF, 30, 0x1249, 0x1249, 30, 0x124E, 0x124F, 30, + 0x1257, 0x1257, 30, 0x1259, 0x1259, 30, 0x125E, 0x125F, 30, 0x1289, 0x1289, 30, + 0x128E, 0x128F, 30, 0x12B1, 0x12B1, 30, 0x12B6, 0x12B7, 30, 0x12BF, 0x12BF, 30, + 0x12C1, 0x12C1, 30, 0x12C6, 0x12C7, 30, 0x12D7, 0x12D7, 30, 0x1311, 0x1311, 30, + 0x1316, 0x1317, 30, 0x135B, 0x135C, 30, 0x137D, 0x137F, 30, 0x139A, 0x139F, 30, + 0x13F6, 0x13F7, 30, 0x13FE, 0x13FF, 30, 0x169D, 0x169F, 30, 0x16F9, 0x16FF, 30, + 0x1716, 0x171E, 30, 0x1737, 0x173F, 30, 0x1754, 0x175F, 30, 0x176D, 0x176D, 30, + 0x1771, 0x1771, 30, 0x1774, 0x177F, 30, 0x17DE, 0x17DF, 30, 0x17EA, 0x17EF, 30, + 0x17FA, 0x17FF, 30, 0x181A, 0x181F, 30, 0x1879, 0x187F, 30, 0x18AB, 0x18AF, 30, + 0x18F6, 0x18FF, 30, 0x191F, 0x191F, 30, 0x192C, 0x192F, 30, 0x193C, 0x193F, 30, + 0x1941, 0x1943, 30, 0x196E, 0x196F, 30, 0x1975, 0x197F, 30, 0x19AC, 0x19AF, 30, + 0x19CA, 0x19CF, 30, 0x19DB, 0x19DD, 30, 0x1A1C, 0x1A1D, 30, 0x1A5F, 0x1A5F, 30, + 0x1A7D, 0x1A7E, 30, 0x1A8A, 0x1A8F, 30, 0x1A9A, 0x1A9F, 30, 0x1AAE, 0x1AAF, 30, + 0x1ADE, 0x1ADF, 30, 0x1AEC, 0x1AFF, 30, 0x1B4D, 0x1B4D, 30, 0x1BF4, 0x1BFB, 30, + 0x1C38, 0x1C3A, 30, 0x1C4A, 0x1C4C, 30, 0x1C8B, 0x1C8F, 30, 0x1CBB, 0x1CBC, 30, + 0x1CC8, 0x1CCF, 30, 0x1CFB, 0x1CFF, 30, 0x1F16, 0x1F17, 30, 0x1F1E, 0x1F1F, 30, + 0x1F46, 0x1F47, 30, 0x1F4E, 0x1F4F, 30, 0x1F58, 0x1F58, 30, 0x1F5A, 0x1F5A, 30, + 0x1F5C, 0x1F5C, 30, 0x1F5E, 0x1F5E, 30, 0x1F7E, 0x1F7F, 30, 0x1FB5, 0x1FB5, 30, + 0x1FC5, 0x1FC5, 30, 0x1FD4, 0x1FD5, 30, 0x1FDC, 0x1FDC, 30, 0x1FF0, 0x1FF1, 30, + 0x1FF5, 0x1FF5, 30, 0x1FFF, 0x1FFF, 30, 0x2065, 0x2065, 30, 0x2072, 0x2073, 30, + 0x208F, 0x208F, 30, 0x209D, 0x209F, 30, 0x20C2, 0x20CF, 30, 0x20F1, 0x20FF, 30, + 0x218C, 0x218F, 30, 0x242A, 0x243F, 30, 0x244B, 0x245F, 30, 0x2B74, 0x2B75, 30, + 0x2CF4, 0x2CF8, 30, 0x2D26, 0x2D26, 30, 0x2D28, 0x2D2C, 30, 0x2D2E, 0x2D2F, 30, + 0x2D68, 0x2D6E, 30, 0x2D71, 0x2D7E, 30, 0x2D97, 0x2D9F, 30, 0x2DA7, 0x2DA7, 30, + }; + } + + private static int[] rangeChunk11() { + return new int[] { + 0x2DAF, 0x2DAF, 30, 0x2DB7, 0x2DB7, 30, 0x2DBF, 0x2DBF, 30, 0x2DC7, 0x2DC7, 30, + 0x2DCF, 0x2DCF, 30, 0x2DD7, 0x2DD7, 30, 0x2DDF, 0x2DDF, 30, 0x2E5E, 0x2E7F, 30, + 0x2E9A, 0x2E9A, 30, 0x2EF4, 0x2EFF, 30, 0x2FD6, 0x2FEF, 30, 0x3040, 0x3040, 30, + 0x3097, 0x3098, 30, 0x3100, 0x3104, 30, 0x3130, 0x3130, 30, 0x318F, 0x318F, 30, + 0x31E6, 0x31EE, 30, 0x321F, 0x321F, 30, 0xA48D, 0xA48F, 30, 0xA4C7, 0xA4CF, 30, + 0xA62C, 0xA63F, 30, 0xA6F8, 0xA6FF, 30, 0xA7DD, 0xA7F0, 30, 0xA82D, 0xA82F, 30, + 0xA83A, 0xA83F, 30, 0xA878, 0xA87F, 30, 0xA8C6, 0xA8CD, 30, 0xA8DA, 0xA8DF, 30, + 0xA954, 0xA95E, 30, 0xA97D, 0xA97F, 30, 0xA9CE, 0xA9CE, 30, 0xA9DA, 0xA9DD, 30, + 0xA9FF, 0xA9FF, 30, 0xAA37, 0xAA3F, 30, 0xAA4E, 0xAA4F, 30, 0xAA5A, 0xAA5B, 30, + 0xAAC3, 0xAADA, 30, 0xAAF7, 0xAB00, 30, 0xAB07, 0xAB08, 30, 0xAB0F, 0xAB10, 30, + 0xAB17, 0xAB1F, 30, 0xAB27, 0xAB27, 30, 0xAB2F, 0xAB2F, 30, 0xAB6C, 0xAB6F, 30, + 0xABEE, 0xABEF, 30, 0xABFA, 0xABFF, 30, 0xD7A4, 0xD7AF, 30, 0xD7C7, 0xD7CA, 30, + 0xD7FC, 0xD7FF, 30, 0xFA6E, 0xFA6F, 30, 0xFADA, 0xFAFF, 30, 0xFB07, 0xFB12, 30, + 0xFB18, 0xFB1C, 30, 0xFB37, 0xFB37, 30, 0xFB3D, 0xFB3D, 30, 0xFB3F, 0xFB3F, 30, + 0xFB42, 0xFB42, 30, 0xFB45, 0xFB45, 30, 0xFDD0, 0xFDEF, 30, 0xFE1A, 0xFE1F, 30, + 0xFE53, 0xFE53, 30, 0xFE67, 0xFE67, 30, 0xFE6C, 0xFE6F, 30, 0xFE75, 0xFE75, 30, + 0xFEFD, 0xFEFE, 30, 0xFF00, 0xFF00, 30, 0xFFBF, 0xFFC1, 30, 0xFFC8, 0xFFC9, 30, + 0xFFD0, 0xFFD1, 30, 0xFFD8, 0xFFD9, 30, 0xFFDD, 0xFFDF, 30, 0xFFE7, 0xFFE7, 30, + 0xFFEF, 0xFFF8, 30, 0xFFFE, 0xFFFF, 30, 0x1000C, 0x1000C, 30, 0x10027, 0x10027, 30, + 0x1003B, 0x1003B, 30, 0x1003E, 0x1003E, 30, 0x1004E, 0x1004F, 30, 0x1005E, 0x1007F, 30, + 0x100FB, 0x100FF, 30, 0x10103, 0x10106, 30, 0x10134, 0x10136, 30, 0x1018F, 0x1018F, 30, + 0x1019D, 0x1019F, 30, 0x101A1, 0x101CF, 30, 0x101FE, 0x1027F, 30, 0x1029D, 0x1029F, 30, + 0x102D1, 0x102DF, 30, 0x102FC, 0x102FF, 30, 0x10324, 0x1032C, 30, 0x1034B, 0x1034F, 30, + 0x1037B, 0x1037F, 30, 0x1039E, 0x1039E, 30, 0x103C4, 0x103C7, 30, 0x103D6, 0x103FF, 30, + 0x1049E, 0x1049F, 30, 0x104AA, 0x104AF, 30, 0x104D4, 0x104D7, 30, 0x104FC, 0x104FF, 30, + 0x10528, 0x1052F, 30, 0x10564, 0x1056E, 30, 0x1057B, 0x1057B, 30, 0x1058B, 0x1058B, 30, + 0x10593, 0x10593, 30, 0x10596, 0x10596, 30, 0x105A2, 0x105A2, 30, 0x105B2, 0x105B2, 30, + 0x105BA, 0x105BA, 30, 0x105BD, 0x105BF, 30, 0x105F4, 0x105FF, 30, 0x10737, 0x1073F, 30, + 0x10756, 0x1075F, 30, 0x10768, 0x1077F, 30, 0x10786, 0x10786, 30, 0x107B1, 0x107B1, 30, + 0x107BB, 0x107FF, 30, 0x10806, 0x10807, 30, 0x10809, 0x10809, 30, 0x10836, 0x10836, 30, + 0x10839, 0x1083B, 30, 0x1083D, 0x1083E, 30, 0x10856, 0x10856, 30, 0x1089F, 0x108A6, 30, + 0x108B0, 0x108DF, 30, 0x108F3, 0x108F3, 30, 0x108F6, 0x108FA, 30, 0x1091C, 0x1091E, 30, + 0x1093A, 0x1093E, 30, 0x1095A, 0x1097F, 30, 0x109B8, 0x109BB, 30, 0x109D0, 0x109D1, 30, + 0x10A04, 0x10A04, 30, 0x10A07, 0x10A0B, 30, 0x10A14, 0x10A14, 30, 0x10A18, 0x10A18, 30, + 0x10A36, 0x10A37, 30, 0x10A3B, 0x10A3E, 30, 0x10A49, 0x10A4F, 30, 0x10A59, 0x10A5F, 30, + 0x10AA0, 0x10ABF, 30, 0x10AE7, 0x10AEA, 30, 0x10AF7, 0x10AFF, 30, 0x10B36, 0x10B38, 30, + 0x10B56, 0x10B57, 30, 0x10B73, 0x10B77, 30, 0x10B92, 0x10B98, 30, 0x10B9D, 0x10BA8, 30, + 0x10BB0, 0x10BFF, 30, 0x10C49, 0x10C7F, 30, 0x10CB3, 0x10CBF, 30, 0x10CF3, 0x10CF9, 30, + 0x10D28, 0x10D2F, 30, 0x10D3A, 0x10D3F, 30, 0x10D66, 0x10D68, 30, 0x10D86, 0x10D8D, 30, + 0x10D90, 0x10E5F, 30, 0x10E7F, 0x10E7F, 30, 0x10EAA, 0x10EAA, 30, 0x10EAE, 0x10EAF, 30, + 0x10EB2, 0x10EC1, 30, 0x10EC8, 0x10ECF, 30, 0x10ED9, 0x10EF9, 30, 0x10F28, 0x10F2F, 30, + 0x10F5A, 0x10F6F, 30, 0x10F8A, 0x10FAF, 30, 0x10FCC, 0x10FDF, 30, 0x10FF7, 0x10FFF, 30, + 0x1104E, 0x11051, 30, 0x11076, 0x1107E, 30, 0x110C3, 0x110CC, 30, 0x110CE, 0x110CF, 30, + 0x110E9, 0x110EF, 30, 0x110FA, 0x110FF, 30, 0x11135, 0x11135, 30, 0x11148, 0x1114F, 30, + 0x11177, 0x1117F, 30, 0x111E0, 0x111E0, 30, 0x111F5, 0x111FF, 30, 0x11212, 0x11212, 30, + 0x11242, 0x1127F, 30, 0x11287, 0x11287, 30, 0x11289, 0x11289, 30, 0x1128E, 0x1128E, 30, + 0x1129E, 0x1129E, 30, 0x112AA, 0x112AF, 30, 0x112EB, 0x112EF, 30, 0x112FA, 0x112FF, 30, + 0x11304, 0x11304, 30, 0x1130D, 0x1130E, 30, 0x11311, 0x11312, 30, 0x11329, 0x11329, 30, + 0x11331, 0x11331, 30, 0x11334, 0x11334, 30, 0x1133A, 0x1133A, 30, 0x11345, 0x11346, 30, + 0x11349, 0x1134A, 30, 0x1134E, 0x1134F, 30, 0x11351, 0x11356, 30, 0x11358, 0x1135C, 30, + 0x11364, 0x11365, 30, 0x1136D, 0x1136F, 30, 0x11375, 0x1137F, 30, 0x1138A, 0x1138A, 30, + 0x1138C, 0x1138D, 30, 0x1138F, 0x1138F, 30, 0x113B6, 0x113B6, 30, 0x113C1, 0x113C1, 30, + 0x113C3, 0x113C4, 30, 0x113C6, 0x113C6, 30, 0x113CB, 0x113CB, 30, 0x113D6, 0x113D6, 30, + 0x113D9, 0x113E0, 30, 0x113E3, 0x113FF, 30, 0x1145C, 0x1145C, 30, 0x11462, 0x1147F, 30, + 0x114C8, 0x114CF, 30, 0x114DA, 0x1157F, 30, 0x115B6, 0x115B7, 30, 0x115DE, 0x115FF, 30, + 0x11645, 0x1164F, 30, 0x1165A, 0x1165F, 30, 0x1166D, 0x1167F, 30, 0x116BA, 0x116BF, 30, + 0x116CA, 0x116CF, 30, 0x116E4, 0x116FF, 30, 0x1171B, 0x1171C, 30, 0x1172C, 0x1172F, 30, + 0x11747, 0x117FF, 30, 0x1183C, 0x1189F, 30, 0x118F3, 0x118FE, 30, 0x11907, 0x11908, 30, + 0x1190A, 0x1190B, 30, 0x11914, 0x11914, 30, 0x11917, 0x11917, 30, 0x11936, 0x11936, 30, + 0x11939, 0x1193A, 30, 0x11947, 0x1194F, 30, 0x1195A, 0x1199F, 30, 0x119A8, 0x119A9, 30, + 0x119D8, 0x119D9, 30, 0x119E5, 0x119FF, 30, 0x11A48, 0x11A4F, 30, 0x11AA3, 0x11AAF, 30, + 0x11AF9, 0x11AFF, 30, 0x11B0A, 0x11B5F, 30, 0x11B68, 0x11BBF, 30, 0x11BE2, 0x11BEF, 30, + 0x11BFA, 0x11BFF, 30, 0x11C09, 0x11C09, 30, 0x11C37, 0x11C37, 30, 0x11C46, 0x11C4F, 30, + 0x11C6D, 0x11C6F, 30, 0x11C90, 0x11C91, 30, 0x11CA8, 0x11CA8, 30, 0x11CB7, 0x11CFF, 30, + 0x11D07, 0x11D07, 30, 0x11D0A, 0x11D0A, 30, 0x11D37, 0x11D39, 30, 0x11D3B, 0x11D3B, 30, + 0x11D3E, 0x11D3E, 30, 0x11D48, 0x11D4F, 30, 0x11D5A, 0x11D5F, 30, 0x11D66, 0x11D66, 30, + 0x11D69, 0x11D69, 30, 0x11D8F, 0x11D8F, 30, 0x11D92, 0x11D92, 30, 0x11D99, 0x11D9F, 30, + 0x11DAA, 0x11DAF, 30, 0x11DDC, 0x11DDF, 30, 0x11DEA, 0x11EDF, 30, 0x11EF9, 0x11EFF, 30, + 0x11F11, 0x11F11, 30, 0x11F3B, 0x11F3D, 30, 0x11F5B, 0x11FAF, 30, 0x11FB1, 0x11FBF, 30, + 0x11FF2, 0x11FFE, 30, 0x1239A, 0x123FF, 30, 0x1246F, 0x1246F, 30, 0x12475, 0x1247F, 30, + 0x12544, 0x12F8F, 30, 0x12FF3, 0x12FFF, 30, 0x13456, 0x1345F, 30, 0x143FB, 0x143FF, 30, + 0x14647, 0x160FF, 30, 0x1613A, 0x167FF, 30, 0x16A39, 0x16A3F, 30, 0x16A5F, 0x16A5F, 30, + 0x16A6A, 0x16A6D, 30, 0x16ABF, 0x16ABF, 30, 0x16ACA, 0x16ACF, 30, 0x16AEE, 0x16AEF, 30, + 0x16AF6, 0x16AFF, 30, 0x16B46, 0x16B4F, 30, 0x16B5A, 0x16B5A, 30, 0x16B62, 0x16B62, 30, + 0x16B78, 0x16B7C, 30, 0x16B90, 0x16D3F, 30, 0x16D7A, 0x16E3F, 30, 0x16E9B, 0x16E9F, 30, + 0x16EB9, 0x16EBA, 30, 0x16ED4, 0x16EFF, 30, 0x16F4B, 0x16F4E, 30, 0x16F88, 0x16F8E, 30, + 0x16FA0, 0x16FDF, 30, 0x16FE5, 0x16FEF, 30, 0x16FF7, 0x16FFF, 30, 0x18CD6, 0x18CFE, 30, + 0x18D1F, 0x18D7F, 30, 0x18DF3, 0x1AFEF, 30, 0x1AFF4, 0x1AFF4, 30, 0x1AFFC, 0x1AFFC, 30, + 0x1AFFF, 0x1AFFF, 30, 0x1B123, 0x1B131, 30, 0x1B133, 0x1B14F, 30, 0x1B153, 0x1B154, 30, + 0x1B156, 0x1B163, 30, 0x1B168, 0x1B16F, 30, 0x1B2FC, 0x1BBFF, 30, 0x1BC6B, 0x1BC6F, 30, + 0x1BC7D, 0x1BC7F, 30, 0x1BC89, 0x1BC8F, 30, 0x1BC9A, 0x1BC9B, 30, 0x1BCA4, 0x1CBFF, 30, + 0x1CCFD, 0x1CCFF, 30, 0x1CEB4, 0x1CEB9, 30, 0x1CED1, 0x1CEDF, 30, 0x1CEF1, 0x1CEFF, 30, + 0x1CF2E, 0x1CF2F, 30, 0x1CF47, 0x1CF4F, 30, 0x1CFC4, 0x1CFFF, 30, 0x1D0F6, 0x1D0FF, 30, + 0x1D127, 0x1D128, 30, 0x1D1EB, 0x1D1FF, 30, 0x1D246, 0x1D2BF, 30, 0x1D2D4, 0x1D2DF, 30, + 0x1D2F4, 0x1D2FF, 30, 0x1D357, 0x1D35F, 30, 0x1D379, 0x1D3FF, 30, 0x1D455, 0x1D455, 30, + 0x1D49D, 0x1D49D, 30, 0x1D4A0, 0x1D4A1, 30, 0x1D4A3, 0x1D4A4, 30, 0x1D4A7, 0x1D4A8, 30, + 0x1D4AD, 0x1D4AD, 30, 0x1D4BA, 0x1D4BA, 30, 0x1D4BC, 0x1D4BC, 30, 0x1D4C4, 0x1D4C4, 30, + 0x1D506, 0x1D506, 30, 0x1D50B, 0x1D50C, 30, 0x1D515, 0x1D515, 30, 0x1D51D, 0x1D51D, 30, + 0x1D53A, 0x1D53A, 30, 0x1D53F, 0x1D53F, 30, 0x1D545, 0x1D545, 30, 0x1D547, 0x1D549, 30, + 0x1D551, 0x1D551, 30, 0x1D6A6, 0x1D6A7, 30, 0x1D7CC, 0x1D7CD, 30, 0x1DA8C, 0x1DA9A, 30, + 0x1DAA0, 0x1DAA0, 30, 0x1DAB0, 0x1DEFF, 30, 0x1DF1F, 0x1DF24, 30, 0x1DF2B, 0x1DFFF, 30, + 0x1E007, 0x1E007, 30, 0x1E019, 0x1E01A, 30, 0x1E022, 0x1E022, 30, 0x1E025, 0x1E025, 30, + 0x1E02B, 0x1E02F, 30, 0x1E06E, 0x1E08E, 30, 0x1E090, 0x1E0FF, 30, 0x1E12D, 0x1E12F, 30, + 0x1E13E, 0x1E13F, 30, 0x1E14A, 0x1E14D, 30, 0x1E150, 0x1E28F, 30, 0x1E2AF, 0x1E2BF, 30, + 0x1E2FA, 0x1E2FE, 30, 0x1E300, 0x1E4CF, 30, 0x1E4FA, 0x1E5CF, 30, 0x1E5FB, 0x1E5FE, 30, + 0x1E600, 0x1E6BF, 30, 0x1E6DF, 0x1E6DF, 30, 0x1E6F6, 0x1E6FD, 30, 0x1E700, 0x1E7DF, 30, + 0x1E7E7, 0x1E7E7, 30, 0x1E7EC, 0x1E7EC, 30, 0x1E7EF, 0x1E7EF, 30, 0x1E7FF, 0x1E7FF, 30, + 0x1E8C5, 0x1E8C6, 30, 0x1E8D7, 0x1E8FF, 30, 0x1E94C, 0x1E94F, 30, 0x1E95A, 0x1E95D, 30, + 0x1E960, 0x1EC70, 30, 0x1ECB5, 0x1ED00, 30, 0x1ED3E, 0x1EDFF, 30, 0x1EE04, 0x1EE04, 30, + 0x1EE20, 0x1EE20, 30, 0x1EE23, 0x1EE23, 30, 0x1EE25, 0x1EE26, 30, 0x1EE28, 0x1EE28, 30, + }; + } + + private static int[] rangeChunk12() { + return new int[] { + 0x1EE33, 0x1EE33, 30, 0x1EE38, 0x1EE38, 30, 0x1EE3A, 0x1EE3A, 30, 0x1EE3C, 0x1EE41, 30, + 0x1EE43, 0x1EE46, 30, 0x1EE48, 0x1EE48, 30, 0x1EE4A, 0x1EE4A, 30, 0x1EE4C, 0x1EE4C, 30, + 0x1EE50, 0x1EE50, 30, 0x1EE53, 0x1EE53, 30, 0x1EE55, 0x1EE56, 30, 0x1EE58, 0x1EE58, 30, + 0x1EE5A, 0x1EE5A, 30, 0x1EE5C, 0x1EE5C, 30, 0x1EE5E, 0x1EE5E, 30, 0x1EE60, 0x1EE60, 30, + 0x1EE63, 0x1EE63, 30, 0x1EE65, 0x1EE66, 30, 0x1EE6B, 0x1EE6B, 30, 0x1EE73, 0x1EE73, 30, + 0x1EE78, 0x1EE78, 30, 0x1EE7D, 0x1EE7D, 30, 0x1EE7F, 0x1EE7F, 30, 0x1EE8A, 0x1EE8A, 30, + 0x1EE9C, 0x1EEA0, 30, 0x1EEA4, 0x1EEA4, 30, 0x1EEAA, 0x1EEAA, 30, 0x1EEBC, 0x1EEEF, 30, + 0x1EEF2, 0x1EFFF, 30, 0x1F02C, 0x1F02F, 30, 0x1F094, 0x1F09F, 30, 0x1F0AF, 0x1F0B0, 30, + 0x1F0C0, 0x1F0C0, 30, 0x1F0D0, 0x1F0D0, 30, 0x1F0F6, 0x1F0FF, 30, 0x1F1AE, 0x1F1E5, 30, + 0x1F203, 0x1F20F, 30, 0x1F23C, 0x1F23F, 30, 0x1F249, 0x1F24F, 30, 0x1F252, 0x1F25F, 30, + 0x1F266, 0x1F2FF, 30, 0x1F6D9, 0x1F6DB, 30, 0x1F6ED, 0x1F6EF, 30, 0x1F6FD, 0x1F6FF, 30, + 0x1F7DA, 0x1F7DF, 30, 0x1F7EC, 0x1F7EF, 30, 0x1F7F1, 0x1F7FF, 30, 0x1F80C, 0x1F80F, 30, + 0x1F848, 0x1F84F, 30, 0x1F85A, 0x1F85F, 30, 0x1F888, 0x1F88F, 30, 0x1F8AE, 0x1F8AF, 30, + 0x1F8BC, 0x1F8BF, 30, 0x1F8C2, 0x1F8CF, 30, 0x1F8D9, 0x1F8FF, 30, 0x1FA58, 0x1FA5F, 30, + 0x1FA6E, 0x1FA6F, 30, 0x1FA7D, 0x1FA7F, 30, 0x1FA8B, 0x1FA8D, 30, 0x1FAC7, 0x1FAC7, 30, + 0x1FAC9, 0x1FACC, 30, 0x1FADD, 0x1FADE, 30, 0x1FAEB, 0x1FAEE, 30, 0x1FAF9, 0x1FAFF, 30, + 0x1FB93, 0x1FB93, 30, 0x1FBFB, 0x1FFFF, 30, 0x2A6E0, 0x2A6FF, 30, 0x2B81E, 0x2B81F, 30, + 0x2CEAE, 0x2CEAF, 30, 0x2EBE1, 0x2EBEF, 30, 0x2EE5E, 0x2F7FF, 30, 0x2FA1E, 0x2FFFF, 30, + 0x3134B, 0x3134F, 30, 0x3347A, 0xE0000, 30, 0xE0002, 0xE001F, 30, 0xE0080, 0xE00FF, 30, + 0xE01F0, 0xEFFFF, 30, 0xFFFFE, 0xFFFFF, 30, 0x10FFFE, 0x10FFFF, 30, 0x0, 0x1F, 30, + 0x7F, 0x9F, 30, 0xAD, 0xAD, 30, 0x600, 0x605, 30, 0x61C, 0x61C, 30, + 0x6DD, 0x6DD, 30, 0x70F, 0x70F, 30, 0x890, 0x891, 30, 0x8E2, 0x8E2, 30, + 0x180E, 0x180E, 30, 0x200B, 0x200F, 30, 0x202A, 0x202E, 30, 0x2060, 0x2064, 30, + 0x2066, 0x206F, 30, 0xFEFF, 0xFEFF, 30, 0xFFF9, 0xFFFB, 30, 0x110BD, 0x110BD, 30, + 0x110CD, 0x110CD, 30, 0x13430, 0x1343F, 30, 0x1BCA0, 0x1BCA3, 30, 0x1D173, 0x1D17A, 30, + 0xE0001, 0xE0001, 30, 0xE0020, 0xE007F, 30, 0xE000, 0xF8FF, 30, 0xF0000, 0xFFFFD, 30, + 0x100000, 0x10FFFD, 30, 0xD800, 0xDFFF, 30, 0x41, 0x5A, 31, 0xC0, 0xD6, 31, + 0xD8, 0xDE, 31, 0x100, 0x100, 31, 0x102, 0x102, 31, 0x104, 0x104, 31, + 0x106, 0x106, 31, 0x108, 0x108, 31, 0x10A, 0x10A, 31, 0x10C, 0x10C, 31, + 0x10E, 0x10E, 31, 0x110, 0x110, 31, 0x112, 0x112, 31, 0x114, 0x114, 31, + 0x116, 0x116, 31, 0x118, 0x118, 31, 0x11A, 0x11A, 31, 0x11C, 0x11C, 31, + 0x11E, 0x11E, 31, 0x120, 0x120, 31, 0x122, 0x122, 31, 0x124, 0x124, 31, + 0x126, 0x126, 31, 0x128, 0x128, 31, 0x12A, 0x12A, 31, 0x12C, 0x12C, 31, + 0x12E, 0x12E, 31, 0x130, 0x130, 31, 0x132, 0x132, 31, 0x134, 0x134, 31, + 0x136, 0x136, 31, 0x139, 0x139, 31, 0x13B, 0x13B, 31, 0x13D, 0x13D, 31, + 0x13F, 0x13F, 31, 0x141, 0x141, 31, 0x143, 0x143, 31, 0x145, 0x145, 31, + 0x147, 0x147, 31, 0x14A, 0x14A, 31, 0x14C, 0x14C, 31, 0x14E, 0x14E, 31, + 0x150, 0x150, 31, 0x152, 0x152, 31, 0x154, 0x154, 31, 0x156, 0x156, 31, + 0x158, 0x158, 31, 0x15A, 0x15A, 31, 0x15C, 0x15C, 31, 0x15E, 0x15E, 31, + 0x160, 0x160, 31, 0x162, 0x162, 31, 0x164, 0x164, 31, 0x166, 0x166, 31, + 0x168, 0x168, 31, 0x16A, 0x16A, 31, 0x16C, 0x16C, 31, 0x16E, 0x16E, 31, + 0x170, 0x170, 31, 0x172, 0x172, 31, 0x174, 0x174, 31, 0x176, 0x176, 31, + 0x178, 0x179, 31, 0x17B, 0x17B, 31, 0x17D, 0x17D, 31, 0x181, 0x182, 31, + 0x184, 0x184, 31, 0x186, 0x187, 31, 0x189, 0x18B, 31, 0x18E, 0x191, 31, + 0x193, 0x194, 31, 0x196, 0x198, 31, 0x19C, 0x19D, 31, 0x19F, 0x1A0, 31, + 0x1A2, 0x1A2, 31, 0x1A4, 0x1A4, 31, 0x1A6, 0x1A7, 31, 0x1A9, 0x1A9, 31, + 0x1AC, 0x1AC, 31, 0x1AE, 0x1AF, 31, 0x1B1, 0x1B3, 31, 0x1B5, 0x1B5, 31, + 0x1B7, 0x1B8, 31, 0x1BC, 0x1BC, 31, 0x1C4, 0x1C4, 31, 0x1C7, 0x1C7, 31, + 0x1CA, 0x1CA, 31, 0x1CD, 0x1CD, 31, 0x1CF, 0x1CF, 31, 0x1D1, 0x1D1, 31, + 0x1D3, 0x1D3, 31, 0x1D5, 0x1D5, 31, 0x1D7, 0x1D7, 31, 0x1D9, 0x1D9, 31, + 0x1DB, 0x1DB, 31, 0x1DE, 0x1DE, 31, 0x1E0, 0x1E0, 31, 0x1E2, 0x1E2, 31, + 0x1E4, 0x1E4, 31, 0x1E6, 0x1E6, 31, 0x1E8, 0x1E8, 31, 0x1EA, 0x1EA, 31, + 0x1EC, 0x1EC, 31, 0x1EE, 0x1EE, 31, 0x1F1, 0x1F1, 31, 0x1F4, 0x1F4, 31, + 0x1F6, 0x1F8, 31, 0x1FA, 0x1FA, 31, 0x1FC, 0x1FC, 31, 0x1FE, 0x1FE, 31, + 0x200, 0x200, 31, 0x202, 0x202, 31, 0x204, 0x204, 31, 0x206, 0x206, 31, + 0x208, 0x208, 31, 0x20A, 0x20A, 31, 0x20C, 0x20C, 31, 0x20E, 0x20E, 31, + 0x210, 0x210, 31, 0x212, 0x212, 31, 0x214, 0x214, 31, 0x216, 0x216, 31, + 0x218, 0x218, 31, 0x21A, 0x21A, 31, 0x21C, 0x21C, 31, 0x21E, 0x21E, 31, + 0x220, 0x220, 31, 0x222, 0x222, 31, 0x224, 0x224, 31, 0x226, 0x226, 31, + 0x228, 0x228, 31, 0x22A, 0x22A, 31, 0x22C, 0x22C, 31, 0x22E, 0x22E, 31, + 0x230, 0x230, 31, 0x232, 0x232, 31, 0x23A, 0x23B, 31, 0x23D, 0x23E, 31, + 0x241, 0x241, 31, 0x243, 0x246, 31, 0x248, 0x248, 31, 0x24A, 0x24A, 31, + 0x24C, 0x24C, 31, 0x24E, 0x24E, 31, 0x370, 0x370, 31, 0x372, 0x372, 31, + 0x376, 0x376, 31, 0x37F, 0x37F, 31, 0x386, 0x386, 31, 0x388, 0x38A, 31, + 0x38C, 0x38C, 31, 0x38E, 0x38F, 31, 0x391, 0x3A1, 31, 0x3A3, 0x3AB, 31, + 0x3CF, 0x3CF, 31, 0x3D2, 0x3D4, 31, 0x3D8, 0x3D8, 31, 0x3DA, 0x3DA, 31, + 0x3DC, 0x3DC, 31, 0x3DE, 0x3DE, 31, 0x3E0, 0x3E0, 31, 0x3E2, 0x3E2, 31, + 0x3E4, 0x3E4, 31, 0x3E6, 0x3E6, 31, 0x3E8, 0x3E8, 31, 0x3EA, 0x3EA, 31, + 0x3EC, 0x3EC, 31, 0x3EE, 0x3EE, 31, 0x3F4, 0x3F4, 31, 0x3F7, 0x3F7, 31, + 0x3F9, 0x3FA, 31, 0x3FD, 0x42F, 31, 0x460, 0x460, 31, 0x462, 0x462, 31, + 0x464, 0x464, 31, 0x466, 0x466, 31, 0x468, 0x468, 31, 0x46A, 0x46A, 31, + 0x46C, 0x46C, 31, 0x46E, 0x46E, 31, 0x470, 0x470, 31, 0x472, 0x472, 31, + 0x474, 0x474, 31, 0x476, 0x476, 31, 0x478, 0x478, 31, 0x47A, 0x47A, 31, + 0x47C, 0x47C, 31, 0x47E, 0x47E, 31, 0x480, 0x480, 31, 0x48A, 0x48A, 31, + 0x48C, 0x48C, 31, 0x48E, 0x48E, 31, 0x490, 0x490, 31, 0x492, 0x492, 31, + 0x494, 0x494, 31, 0x496, 0x496, 31, 0x498, 0x498, 31, 0x49A, 0x49A, 31, + 0x49C, 0x49C, 31, 0x49E, 0x49E, 31, 0x4A0, 0x4A0, 31, 0x4A2, 0x4A2, 31, + 0x4A4, 0x4A4, 31, 0x4A6, 0x4A6, 31, 0x4A8, 0x4A8, 31, 0x4AA, 0x4AA, 31, + 0x4AC, 0x4AC, 31, 0x4AE, 0x4AE, 31, 0x4B0, 0x4B0, 31, 0x4B2, 0x4B2, 31, + 0x4B4, 0x4B4, 31, 0x4B6, 0x4B6, 31, 0x4B8, 0x4B8, 31, 0x4BA, 0x4BA, 31, + 0x4BC, 0x4BC, 31, 0x4BE, 0x4BE, 31, 0x4C0, 0x4C1, 31, 0x4C3, 0x4C3, 31, + 0x4C5, 0x4C5, 31, 0x4C7, 0x4C7, 31, 0x4C9, 0x4C9, 31, 0x4CB, 0x4CB, 31, + 0x4CD, 0x4CD, 31, 0x4D0, 0x4D0, 31, 0x4D2, 0x4D2, 31, 0x4D4, 0x4D4, 31, + 0x4D6, 0x4D6, 31, 0x4D8, 0x4D8, 31, 0x4DA, 0x4DA, 31, 0x4DC, 0x4DC, 31, + 0x4DE, 0x4DE, 31, 0x4E0, 0x4E0, 31, 0x4E2, 0x4E2, 31, 0x4E4, 0x4E4, 31, + 0x4E6, 0x4E6, 31, 0x4E8, 0x4E8, 31, 0x4EA, 0x4EA, 31, 0x4EC, 0x4EC, 31, + 0x4EE, 0x4EE, 31, 0x4F0, 0x4F0, 31, 0x4F2, 0x4F2, 31, 0x4F4, 0x4F4, 31, + 0x4F6, 0x4F6, 31, 0x4F8, 0x4F8, 31, 0x4FA, 0x4FA, 31, 0x4FC, 0x4FC, 31, + 0x4FE, 0x4FE, 31, 0x500, 0x500, 31, 0x502, 0x502, 31, 0x504, 0x504, 31, + 0x506, 0x506, 31, 0x508, 0x508, 31, 0x50A, 0x50A, 31, 0x50C, 0x50C, 31, + 0x50E, 0x50E, 31, 0x510, 0x510, 31, 0x512, 0x512, 31, 0x514, 0x514, 31, + 0x516, 0x516, 31, 0x518, 0x518, 31, 0x51A, 0x51A, 31, 0x51C, 0x51C, 31, + 0x51E, 0x51E, 31, 0x520, 0x520, 31, 0x522, 0x522, 31, 0x524, 0x524, 31, + 0x526, 0x526, 31, 0x528, 0x528, 31, 0x52A, 0x52A, 31, 0x52C, 0x52C, 31, + 0x52E, 0x52E, 31, 0x531, 0x556, 31, 0x10A0, 0x10C5, 31, 0x10C7, 0x10C7, 31, + 0x10CD, 0x10CD, 31, 0x13A0, 0x13F5, 31, 0x1C89, 0x1C89, 31, 0x1C90, 0x1CBA, 31, + 0x1CBD, 0x1CBF, 31, 0x1E00, 0x1E00, 31, 0x1E02, 0x1E02, 31, 0x1E04, 0x1E04, 31, + 0x1E06, 0x1E06, 31, 0x1E08, 0x1E08, 31, 0x1E0A, 0x1E0A, 31, 0x1E0C, 0x1E0C, 31, + 0x1E0E, 0x1E0E, 31, 0x1E10, 0x1E10, 31, 0x1E12, 0x1E12, 31, 0x1E14, 0x1E14, 31, + 0x1E16, 0x1E16, 31, 0x1E18, 0x1E18, 31, 0x1E1A, 0x1E1A, 31, 0x1E1C, 0x1E1C, 31, + }; + } + + private static int[] rangeChunk13() { + return new int[] { + 0x1E1E, 0x1E1E, 31, 0x1E20, 0x1E20, 31, 0x1E22, 0x1E22, 31, 0x1E24, 0x1E24, 31, + 0x1E26, 0x1E26, 31, 0x1E28, 0x1E28, 31, 0x1E2A, 0x1E2A, 31, 0x1E2C, 0x1E2C, 31, + 0x1E2E, 0x1E2E, 31, 0x1E30, 0x1E30, 31, 0x1E32, 0x1E32, 31, 0x1E34, 0x1E34, 31, + 0x1E36, 0x1E36, 31, 0x1E38, 0x1E38, 31, 0x1E3A, 0x1E3A, 31, 0x1E3C, 0x1E3C, 31, + 0x1E3E, 0x1E3E, 31, 0x1E40, 0x1E40, 31, 0x1E42, 0x1E42, 31, 0x1E44, 0x1E44, 31, + 0x1E46, 0x1E46, 31, 0x1E48, 0x1E48, 31, 0x1E4A, 0x1E4A, 31, 0x1E4C, 0x1E4C, 31, + 0x1E4E, 0x1E4E, 31, 0x1E50, 0x1E50, 31, 0x1E52, 0x1E52, 31, 0x1E54, 0x1E54, 31, + 0x1E56, 0x1E56, 31, 0x1E58, 0x1E58, 31, 0x1E5A, 0x1E5A, 31, 0x1E5C, 0x1E5C, 31, + 0x1E5E, 0x1E5E, 31, 0x1E60, 0x1E60, 31, 0x1E62, 0x1E62, 31, 0x1E64, 0x1E64, 31, + 0x1E66, 0x1E66, 31, 0x1E68, 0x1E68, 31, 0x1E6A, 0x1E6A, 31, 0x1E6C, 0x1E6C, 31, + 0x1E6E, 0x1E6E, 31, 0x1E70, 0x1E70, 31, 0x1E72, 0x1E72, 31, 0x1E74, 0x1E74, 31, + 0x1E76, 0x1E76, 31, 0x1E78, 0x1E78, 31, 0x1E7A, 0x1E7A, 31, 0x1E7C, 0x1E7C, 31, + 0x1E7E, 0x1E7E, 31, 0x1E80, 0x1E80, 31, 0x1E82, 0x1E82, 31, 0x1E84, 0x1E84, 31, + 0x1E86, 0x1E86, 31, 0x1E88, 0x1E88, 31, 0x1E8A, 0x1E8A, 31, 0x1E8C, 0x1E8C, 31, + 0x1E8E, 0x1E8E, 31, 0x1E90, 0x1E90, 31, 0x1E92, 0x1E92, 31, 0x1E94, 0x1E94, 31, + 0x1E9E, 0x1E9E, 31, 0x1EA0, 0x1EA0, 31, 0x1EA2, 0x1EA2, 31, 0x1EA4, 0x1EA4, 31, + 0x1EA6, 0x1EA6, 31, 0x1EA8, 0x1EA8, 31, 0x1EAA, 0x1EAA, 31, 0x1EAC, 0x1EAC, 31, + 0x1EAE, 0x1EAE, 31, 0x1EB0, 0x1EB0, 31, 0x1EB2, 0x1EB2, 31, 0x1EB4, 0x1EB4, 31, + 0x1EB6, 0x1EB6, 31, 0x1EB8, 0x1EB8, 31, 0x1EBA, 0x1EBA, 31, 0x1EBC, 0x1EBC, 31, + 0x1EBE, 0x1EBE, 31, 0x1EC0, 0x1EC0, 31, 0x1EC2, 0x1EC2, 31, 0x1EC4, 0x1EC4, 31, + 0x1EC6, 0x1EC6, 31, 0x1EC8, 0x1EC8, 31, 0x1ECA, 0x1ECA, 31, 0x1ECC, 0x1ECC, 31, + 0x1ECE, 0x1ECE, 31, 0x1ED0, 0x1ED0, 31, 0x1ED2, 0x1ED2, 31, 0x1ED4, 0x1ED4, 31, + 0x1ED6, 0x1ED6, 31, 0x1ED8, 0x1ED8, 31, 0x1EDA, 0x1EDA, 31, 0x1EDC, 0x1EDC, 31, + 0x1EDE, 0x1EDE, 31, 0x1EE0, 0x1EE0, 31, 0x1EE2, 0x1EE2, 31, 0x1EE4, 0x1EE4, 31, + 0x1EE6, 0x1EE6, 31, 0x1EE8, 0x1EE8, 31, 0x1EEA, 0x1EEA, 31, 0x1EEC, 0x1EEC, 31, + 0x1EEE, 0x1EEE, 31, 0x1EF0, 0x1EF0, 31, 0x1EF2, 0x1EF2, 31, 0x1EF4, 0x1EF4, 31, + 0x1EF6, 0x1EF6, 31, 0x1EF8, 0x1EF8, 31, 0x1EFA, 0x1EFA, 31, 0x1EFC, 0x1EFC, 31, + 0x1EFE, 0x1EFE, 31, 0x1F08, 0x1F0F, 31, 0x1F18, 0x1F1D, 31, 0x1F28, 0x1F2F, 31, + 0x1F38, 0x1F3F, 31, 0x1F48, 0x1F4D, 31, 0x1F59, 0x1F59, 31, 0x1F5B, 0x1F5B, 31, + 0x1F5D, 0x1F5D, 31, 0x1F5F, 0x1F5F, 31, 0x1F68, 0x1F6F, 31, 0x1FB8, 0x1FBB, 31, + 0x1FC8, 0x1FCB, 31, 0x1FD8, 0x1FDB, 31, 0x1FE8, 0x1FEC, 31, 0x1FF8, 0x1FFB, 31, + 0x2102, 0x2102, 31, 0x2107, 0x2107, 31, 0x210B, 0x210D, 31, 0x2110, 0x2112, 31, + 0x2115, 0x2115, 31, 0x2119, 0x211D, 31, 0x2124, 0x2124, 31, 0x2126, 0x2126, 31, + 0x2128, 0x2128, 31, 0x212A, 0x212D, 31, 0x2130, 0x2133, 31, 0x213E, 0x213F, 31, + 0x2145, 0x2145, 31, 0x2183, 0x2183, 31, 0x2C00, 0x2C2F, 31, 0x2C60, 0x2C60, 31, + 0x2C62, 0x2C64, 31, 0x2C67, 0x2C67, 31, 0x2C69, 0x2C69, 31, 0x2C6B, 0x2C6B, 31, + 0x2C6D, 0x2C70, 31, 0x2C72, 0x2C72, 31, 0x2C75, 0x2C75, 31, 0x2C7E, 0x2C80, 31, + 0x2C82, 0x2C82, 31, 0x2C84, 0x2C84, 31, 0x2C86, 0x2C86, 31, 0x2C88, 0x2C88, 31, + 0x2C8A, 0x2C8A, 31, 0x2C8C, 0x2C8C, 31, 0x2C8E, 0x2C8E, 31, 0x2C90, 0x2C90, 31, + 0x2C92, 0x2C92, 31, 0x2C94, 0x2C94, 31, 0x2C96, 0x2C96, 31, 0x2C98, 0x2C98, 31, + 0x2C9A, 0x2C9A, 31, 0x2C9C, 0x2C9C, 31, 0x2C9E, 0x2C9E, 31, 0x2CA0, 0x2CA0, 31, + 0x2CA2, 0x2CA2, 31, 0x2CA4, 0x2CA4, 31, 0x2CA6, 0x2CA6, 31, 0x2CA8, 0x2CA8, 31, + 0x2CAA, 0x2CAA, 31, 0x2CAC, 0x2CAC, 31, 0x2CAE, 0x2CAE, 31, 0x2CB0, 0x2CB0, 31, + 0x2CB2, 0x2CB2, 31, 0x2CB4, 0x2CB4, 31, 0x2CB6, 0x2CB6, 31, 0x2CB8, 0x2CB8, 31, + 0x2CBA, 0x2CBA, 31, 0x2CBC, 0x2CBC, 31, 0x2CBE, 0x2CBE, 31, 0x2CC0, 0x2CC0, 31, + 0x2CC2, 0x2CC2, 31, 0x2CC4, 0x2CC4, 31, 0x2CC6, 0x2CC6, 31, 0x2CC8, 0x2CC8, 31, + 0x2CCA, 0x2CCA, 31, 0x2CCC, 0x2CCC, 31, 0x2CCE, 0x2CCE, 31, 0x2CD0, 0x2CD0, 31, + 0x2CD2, 0x2CD2, 31, 0x2CD4, 0x2CD4, 31, 0x2CD6, 0x2CD6, 31, 0x2CD8, 0x2CD8, 31, + 0x2CDA, 0x2CDA, 31, 0x2CDC, 0x2CDC, 31, 0x2CDE, 0x2CDE, 31, 0x2CE0, 0x2CE0, 31, + 0x2CE2, 0x2CE2, 31, 0x2CEB, 0x2CEB, 31, 0x2CED, 0x2CED, 31, 0x2CF2, 0x2CF2, 31, + 0xA640, 0xA640, 31, 0xA642, 0xA642, 31, 0xA644, 0xA644, 31, 0xA646, 0xA646, 31, + 0xA648, 0xA648, 31, 0xA64A, 0xA64A, 31, 0xA64C, 0xA64C, 31, 0xA64E, 0xA64E, 31, + 0xA650, 0xA650, 31, 0xA652, 0xA652, 31, 0xA654, 0xA654, 31, 0xA656, 0xA656, 31, + 0xA658, 0xA658, 31, 0xA65A, 0xA65A, 31, 0xA65C, 0xA65C, 31, 0xA65E, 0xA65E, 31, + 0xA660, 0xA660, 31, 0xA662, 0xA662, 31, 0xA664, 0xA664, 31, 0xA666, 0xA666, 31, + 0xA668, 0xA668, 31, 0xA66A, 0xA66A, 31, 0xA66C, 0xA66C, 31, 0xA680, 0xA680, 31, + 0xA682, 0xA682, 31, 0xA684, 0xA684, 31, 0xA686, 0xA686, 31, 0xA688, 0xA688, 31, + 0xA68A, 0xA68A, 31, 0xA68C, 0xA68C, 31, 0xA68E, 0xA68E, 31, 0xA690, 0xA690, 31, + 0xA692, 0xA692, 31, 0xA694, 0xA694, 31, 0xA696, 0xA696, 31, 0xA698, 0xA698, 31, + 0xA69A, 0xA69A, 31, 0xA722, 0xA722, 31, 0xA724, 0xA724, 31, 0xA726, 0xA726, 31, + 0xA728, 0xA728, 31, 0xA72A, 0xA72A, 31, 0xA72C, 0xA72C, 31, 0xA72E, 0xA72E, 31, + 0xA732, 0xA732, 31, 0xA734, 0xA734, 31, 0xA736, 0xA736, 31, 0xA738, 0xA738, 31, + 0xA73A, 0xA73A, 31, 0xA73C, 0xA73C, 31, 0xA73E, 0xA73E, 31, 0xA740, 0xA740, 31, + 0xA742, 0xA742, 31, 0xA744, 0xA744, 31, 0xA746, 0xA746, 31, 0xA748, 0xA748, 31, + 0xA74A, 0xA74A, 31, 0xA74C, 0xA74C, 31, 0xA74E, 0xA74E, 31, 0xA750, 0xA750, 31, + 0xA752, 0xA752, 31, 0xA754, 0xA754, 31, 0xA756, 0xA756, 31, 0xA758, 0xA758, 31, + 0xA75A, 0xA75A, 31, 0xA75C, 0xA75C, 31, 0xA75E, 0xA75E, 31, 0xA760, 0xA760, 31, + 0xA762, 0xA762, 31, 0xA764, 0xA764, 31, 0xA766, 0xA766, 31, 0xA768, 0xA768, 31, + 0xA76A, 0xA76A, 31, 0xA76C, 0xA76C, 31, 0xA76E, 0xA76E, 31, 0xA779, 0xA779, 31, + 0xA77B, 0xA77B, 31, 0xA77D, 0xA77E, 31, 0xA780, 0xA780, 31, 0xA782, 0xA782, 31, + 0xA784, 0xA784, 31, 0xA786, 0xA786, 31, 0xA78B, 0xA78B, 31, 0xA78D, 0xA78D, 31, + 0xA790, 0xA790, 31, 0xA792, 0xA792, 31, 0xA796, 0xA796, 31, 0xA798, 0xA798, 31, + 0xA79A, 0xA79A, 31, 0xA79C, 0xA79C, 31, 0xA79E, 0xA79E, 31, 0xA7A0, 0xA7A0, 31, + 0xA7A2, 0xA7A2, 31, 0xA7A4, 0xA7A4, 31, 0xA7A6, 0xA7A6, 31, 0xA7A8, 0xA7A8, 31, + 0xA7AA, 0xA7AE, 31, 0xA7B0, 0xA7B4, 31, 0xA7B6, 0xA7B6, 31, 0xA7B8, 0xA7B8, 31, + 0xA7BA, 0xA7BA, 31, 0xA7BC, 0xA7BC, 31, 0xA7BE, 0xA7BE, 31, 0xA7C0, 0xA7C0, 31, + 0xA7C2, 0xA7C2, 31, 0xA7C4, 0xA7C7, 31, 0xA7C9, 0xA7C9, 31, 0xA7CB, 0xA7CC, 31, + 0xA7CE, 0xA7CE, 31, 0xA7D0, 0xA7D0, 31, 0xA7D2, 0xA7D2, 31, 0xA7D4, 0xA7D4, 31, + 0xA7D6, 0xA7D6, 31, 0xA7D8, 0xA7D8, 31, 0xA7DA, 0xA7DA, 31, 0xA7DC, 0xA7DC, 31, + 0xA7F5, 0xA7F5, 31, 0xFF21, 0xFF3A, 31, 0x10400, 0x10427, 31, 0x104B0, 0x104D3, 31, + 0x10570, 0x1057A, 31, 0x1057C, 0x1058A, 31, 0x1058C, 0x10592, 31, 0x10594, 0x10595, 31, + 0x10C80, 0x10CB2, 31, 0x10D50, 0x10D65, 31, 0x118A0, 0x118BF, 31, 0x16E40, 0x16E5F, 31, + 0x16EA0, 0x16EB8, 31, 0x1D400, 0x1D419, 31, 0x1D434, 0x1D44D, 31, 0x1D468, 0x1D481, 31, + 0x1D49C, 0x1D49C, 31, 0x1D49E, 0x1D49F, 31, 0x1D4A2, 0x1D4A2, 31, 0x1D4A5, 0x1D4A6, 31, + 0x1D4A9, 0x1D4AC, 31, 0x1D4AE, 0x1D4B5, 31, 0x1D4D0, 0x1D4E9, 31, 0x1D504, 0x1D505, 31, + 0x1D507, 0x1D50A, 31, 0x1D50D, 0x1D514, 31, 0x1D516, 0x1D51C, 31, 0x1D538, 0x1D539, 31, + 0x1D53B, 0x1D53E, 31, 0x1D540, 0x1D544, 31, 0x1D546, 0x1D546, 31, 0x1D54A, 0x1D550, 31, + 0x1D56C, 0x1D585, 31, 0x1D5A0, 0x1D5B9, 31, 0x1D5D4, 0x1D5ED, 31, 0x1D608, 0x1D621, 31, + 0x1D63C, 0x1D655, 31, 0x1D670, 0x1D689, 31, 0x1D6A8, 0x1D6C0, 31, 0x1D6E2, 0x1D6FA, 31, + 0x1D71C, 0x1D734, 31, 0x1D756, 0x1D76E, 31, 0x1D790, 0x1D7A8, 31, 0x1D7CA, 0x1D7CA, 31, + 0x1E900, 0x1E921, 31, 0x61, 0x7A, 31, 0xB5, 0xB5, 31, 0xDF, 0xF6, 31, + 0xF8, 0xFF, 31, 0x101, 0x101, 31, 0x103, 0x103, 31, 0x105, 0x105, 31, + 0x107, 0x107, 31, 0x109, 0x109, 31, 0x10B, 0x10B, 31, 0x10D, 0x10D, 31, + 0x10F, 0x10F, 31, 0x111, 0x111, 31, 0x113, 0x113, 31, 0x115, 0x115, 31, + 0x117, 0x117, 31, 0x119, 0x119, 31, 0x11B, 0x11B, 31, 0x11D, 0x11D, 31, + 0x11F, 0x11F, 31, 0x121, 0x121, 31, 0x123, 0x123, 31, 0x125, 0x125, 31, + 0x127, 0x127, 31, 0x129, 0x129, 31, 0x12B, 0x12B, 31, 0x12D, 0x12D, 31, + 0x12F, 0x12F, 31, 0x131, 0x131, 31, 0x133, 0x133, 31, 0x135, 0x135, 31, + 0x137, 0x138, 31, 0x13A, 0x13A, 31, 0x13C, 0x13C, 31, 0x13E, 0x13E, 31, + 0x140, 0x140, 31, 0x142, 0x142, 31, 0x144, 0x144, 31, 0x146, 0x146, 31, + }; + } + + private static int[] rangeChunk14() { + return new int[] { + 0x148, 0x149, 31, 0x14B, 0x14B, 31, 0x14D, 0x14D, 31, 0x14F, 0x14F, 31, + 0x151, 0x151, 31, 0x153, 0x153, 31, 0x155, 0x155, 31, 0x157, 0x157, 31, + 0x159, 0x159, 31, 0x15B, 0x15B, 31, 0x15D, 0x15D, 31, 0x15F, 0x15F, 31, + 0x161, 0x161, 31, 0x163, 0x163, 31, 0x165, 0x165, 31, 0x167, 0x167, 31, + 0x169, 0x169, 31, 0x16B, 0x16B, 31, 0x16D, 0x16D, 31, 0x16F, 0x16F, 31, + 0x171, 0x171, 31, 0x173, 0x173, 31, 0x175, 0x175, 31, 0x177, 0x177, 31, + 0x17A, 0x17A, 31, 0x17C, 0x17C, 31, 0x17E, 0x180, 31, 0x183, 0x183, 31, + 0x185, 0x185, 31, 0x188, 0x188, 31, 0x18C, 0x18D, 31, 0x192, 0x192, 31, + 0x195, 0x195, 31, 0x199, 0x19B, 31, 0x19E, 0x19E, 31, 0x1A1, 0x1A1, 31, + 0x1A3, 0x1A3, 31, 0x1A5, 0x1A5, 31, 0x1A8, 0x1A8, 31, 0x1AA, 0x1AB, 31, + 0x1AD, 0x1AD, 31, 0x1B0, 0x1B0, 31, 0x1B4, 0x1B4, 31, 0x1B6, 0x1B6, 31, + 0x1B9, 0x1BA, 31, 0x1BD, 0x1BF, 31, 0x1C6, 0x1C6, 31, 0x1C9, 0x1C9, 31, + 0x1CC, 0x1CC, 31, 0x1CE, 0x1CE, 31, 0x1D0, 0x1D0, 31, 0x1D2, 0x1D2, 31, + 0x1D4, 0x1D4, 31, 0x1D6, 0x1D6, 31, 0x1D8, 0x1D8, 31, 0x1DA, 0x1DA, 31, + 0x1DC, 0x1DD, 31, 0x1DF, 0x1DF, 31, 0x1E1, 0x1E1, 31, 0x1E3, 0x1E3, 31, + 0x1E5, 0x1E5, 31, 0x1E7, 0x1E7, 31, 0x1E9, 0x1E9, 31, 0x1EB, 0x1EB, 31, + 0x1ED, 0x1ED, 31, 0x1EF, 0x1F0, 31, 0x1F3, 0x1F3, 31, 0x1F5, 0x1F5, 31, + 0x1F9, 0x1F9, 31, 0x1FB, 0x1FB, 31, 0x1FD, 0x1FD, 31, 0x1FF, 0x1FF, 31, + 0x201, 0x201, 31, 0x203, 0x203, 31, 0x205, 0x205, 31, 0x207, 0x207, 31, + 0x209, 0x209, 31, 0x20B, 0x20B, 31, 0x20D, 0x20D, 31, 0x20F, 0x20F, 31, + 0x211, 0x211, 31, 0x213, 0x213, 31, 0x215, 0x215, 31, 0x217, 0x217, 31, + 0x219, 0x219, 31, 0x21B, 0x21B, 31, 0x21D, 0x21D, 31, 0x21F, 0x21F, 31, + 0x221, 0x221, 31, 0x223, 0x223, 31, 0x225, 0x225, 31, 0x227, 0x227, 31, + 0x229, 0x229, 31, 0x22B, 0x22B, 31, 0x22D, 0x22D, 31, 0x22F, 0x22F, 31, + 0x231, 0x231, 31, 0x233, 0x239, 31, 0x23C, 0x23C, 31, 0x23F, 0x240, 31, + 0x242, 0x242, 31, 0x247, 0x247, 31, 0x249, 0x249, 31, 0x24B, 0x24B, 31, + 0x24D, 0x24D, 31, 0x24F, 0x293, 31, 0x296, 0x2AF, 31, 0x371, 0x371, 31, + 0x373, 0x373, 31, 0x377, 0x377, 31, 0x37B, 0x37D, 31, 0x390, 0x390, 31, + 0x3AC, 0x3CE, 31, 0x3D0, 0x3D1, 31, 0x3D5, 0x3D7, 31, 0x3D9, 0x3D9, 31, + 0x3DB, 0x3DB, 31, 0x3DD, 0x3DD, 31, 0x3DF, 0x3DF, 31, 0x3E1, 0x3E1, 31, + 0x3E3, 0x3E3, 31, 0x3E5, 0x3E5, 31, 0x3E7, 0x3E7, 31, 0x3E9, 0x3E9, 31, + 0x3EB, 0x3EB, 31, 0x3ED, 0x3ED, 31, 0x3EF, 0x3F3, 31, 0x3F5, 0x3F5, 31, + 0x3F8, 0x3F8, 31, 0x3FB, 0x3FC, 31, 0x430, 0x45F, 31, 0x461, 0x461, 31, + 0x463, 0x463, 31, 0x465, 0x465, 31, 0x467, 0x467, 31, 0x469, 0x469, 31, + 0x46B, 0x46B, 31, 0x46D, 0x46D, 31, 0x46F, 0x46F, 31, 0x471, 0x471, 31, + 0x473, 0x473, 31, 0x475, 0x475, 31, 0x477, 0x477, 31, 0x479, 0x479, 31, + 0x47B, 0x47B, 31, 0x47D, 0x47D, 31, 0x47F, 0x47F, 31, 0x481, 0x481, 31, + 0x48B, 0x48B, 31, 0x48D, 0x48D, 31, 0x48F, 0x48F, 31, 0x491, 0x491, 31, + 0x493, 0x493, 31, 0x495, 0x495, 31, 0x497, 0x497, 31, 0x499, 0x499, 31, + 0x49B, 0x49B, 31, 0x49D, 0x49D, 31, 0x49F, 0x49F, 31, 0x4A1, 0x4A1, 31, + 0x4A3, 0x4A3, 31, 0x4A5, 0x4A5, 31, 0x4A7, 0x4A7, 31, 0x4A9, 0x4A9, 31, + 0x4AB, 0x4AB, 31, 0x4AD, 0x4AD, 31, 0x4AF, 0x4AF, 31, 0x4B1, 0x4B1, 31, + 0x4B3, 0x4B3, 31, 0x4B5, 0x4B5, 31, 0x4B7, 0x4B7, 31, 0x4B9, 0x4B9, 31, + 0x4BB, 0x4BB, 31, 0x4BD, 0x4BD, 31, 0x4BF, 0x4BF, 31, 0x4C2, 0x4C2, 31, + 0x4C4, 0x4C4, 31, 0x4C6, 0x4C6, 31, 0x4C8, 0x4C8, 31, 0x4CA, 0x4CA, 31, + 0x4CC, 0x4CC, 31, 0x4CE, 0x4CF, 31, 0x4D1, 0x4D1, 31, 0x4D3, 0x4D3, 31, + 0x4D5, 0x4D5, 31, 0x4D7, 0x4D7, 31, 0x4D9, 0x4D9, 31, 0x4DB, 0x4DB, 31, + 0x4DD, 0x4DD, 31, 0x4DF, 0x4DF, 31, 0x4E1, 0x4E1, 31, 0x4E3, 0x4E3, 31, + 0x4E5, 0x4E5, 31, 0x4E7, 0x4E7, 31, 0x4E9, 0x4E9, 31, 0x4EB, 0x4EB, 31, + 0x4ED, 0x4ED, 31, 0x4EF, 0x4EF, 31, 0x4F1, 0x4F1, 31, 0x4F3, 0x4F3, 31, + 0x4F5, 0x4F5, 31, 0x4F7, 0x4F7, 31, 0x4F9, 0x4F9, 31, 0x4FB, 0x4FB, 31, + 0x4FD, 0x4FD, 31, 0x4FF, 0x4FF, 31, 0x501, 0x501, 31, 0x503, 0x503, 31, + 0x505, 0x505, 31, 0x507, 0x507, 31, 0x509, 0x509, 31, 0x50B, 0x50B, 31, + 0x50D, 0x50D, 31, 0x50F, 0x50F, 31, 0x511, 0x511, 31, 0x513, 0x513, 31, + 0x515, 0x515, 31, 0x517, 0x517, 31, 0x519, 0x519, 31, 0x51B, 0x51B, 31, + 0x51D, 0x51D, 31, 0x51F, 0x51F, 31, 0x521, 0x521, 31, 0x523, 0x523, 31, + 0x525, 0x525, 31, 0x527, 0x527, 31, 0x529, 0x529, 31, 0x52B, 0x52B, 31, + 0x52D, 0x52D, 31, 0x52F, 0x52F, 31, 0x560, 0x588, 31, 0x10D0, 0x10FA, 31, + 0x10FD, 0x10FF, 31, 0x13F8, 0x13FD, 31, 0x1C80, 0x1C88, 31, 0x1C8A, 0x1C8A, 31, + 0x1D00, 0x1D2B, 31, 0x1D6B, 0x1D77, 31, 0x1D79, 0x1D9A, 31, 0x1E01, 0x1E01, 31, + 0x1E03, 0x1E03, 31, 0x1E05, 0x1E05, 31, 0x1E07, 0x1E07, 31, 0x1E09, 0x1E09, 31, + 0x1E0B, 0x1E0B, 31, 0x1E0D, 0x1E0D, 31, 0x1E0F, 0x1E0F, 31, 0x1E11, 0x1E11, 31, + 0x1E13, 0x1E13, 31, 0x1E15, 0x1E15, 31, 0x1E17, 0x1E17, 31, 0x1E19, 0x1E19, 31, + 0x1E1B, 0x1E1B, 31, 0x1E1D, 0x1E1D, 31, 0x1E1F, 0x1E1F, 31, 0x1E21, 0x1E21, 31, + 0x1E23, 0x1E23, 31, 0x1E25, 0x1E25, 31, 0x1E27, 0x1E27, 31, 0x1E29, 0x1E29, 31, + 0x1E2B, 0x1E2B, 31, 0x1E2D, 0x1E2D, 31, 0x1E2F, 0x1E2F, 31, 0x1E31, 0x1E31, 31, + 0x1E33, 0x1E33, 31, 0x1E35, 0x1E35, 31, 0x1E37, 0x1E37, 31, 0x1E39, 0x1E39, 31, + 0x1E3B, 0x1E3B, 31, 0x1E3D, 0x1E3D, 31, 0x1E3F, 0x1E3F, 31, 0x1E41, 0x1E41, 31, + 0x1E43, 0x1E43, 31, 0x1E45, 0x1E45, 31, 0x1E47, 0x1E47, 31, 0x1E49, 0x1E49, 31, + 0x1E4B, 0x1E4B, 31, 0x1E4D, 0x1E4D, 31, 0x1E4F, 0x1E4F, 31, 0x1E51, 0x1E51, 31, + 0x1E53, 0x1E53, 31, 0x1E55, 0x1E55, 31, 0x1E57, 0x1E57, 31, 0x1E59, 0x1E59, 31, + 0x1E5B, 0x1E5B, 31, 0x1E5D, 0x1E5D, 31, 0x1E5F, 0x1E5F, 31, 0x1E61, 0x1E61, 31, + 0x1E63, 0x1E63, 31, 0x1E65, 0x1E65, 31, 0x1E67, 0x1E67, 31, 0x1E69, 0x1E69, 31, + 0x1E6B, 0x1E6B, 31, 0x1E6D, 0x1E6D, 31, 0x1E6F, 0x1E6F, 31, 0x1E71, 0x1E71, 31, + 0x1E73, 0x1E73, 31, 0x1E75, 0x1E75, 31, 0x1E77, 0x1E77, 31, 0x1E79, 0x1E79, 31, + 0x1E7B, 0x1E7B, 31, 0x1E7D, 0x1E7D, 31, 0x1E7F, 0x1E7F, 31, 0x1E81, 0x1E81, 31, + 0x1E83, 0x1E83, 31, 0x1E85, 0x1E85, 31, 0x1E87, 0x1E87, 31, 0x1E89, 0x1E89, 31, + 0x1E8B, 0x1E8B, 31, 0x1E8D, 0x1E8D, 31, 0x1E8F, 0x1E8F, 31, 0x1E91, 0x1E91, 31, + 0x1E93, 0x1E93, 31, 0x1E95, 0x1E9D, 31, 0x1E9F, 0x1E9F, 31, 0x1EA1, 0x1EA1, 31, + 0x1EA3, 0x1EA3, 31, 0x1EA5, 0x1EA5, 31, 0x1EA7, 0x1EA7, 31, 0x1EA9, 0x1EA9, 31, + 0x1EAB, 0x1EAB, 31, 0x1EAD, 0x1EAD, 31, 0x1EAF, 0x1EAF, 31, 0x1EB1, 0x1EB1, 31, + 0x1EB3, 0x1EB3, 31, 0x1EB5, 0x1EB5, 31, 0x1EB7, 0x1EB7, 31, 0x1EB9, 0x1EB9, 31, + 0x1EBB, 0x1EBB, 31, 0x1EBD, 0x1EBD, 31, 0x1EBF, 0x1EBF, 31, 0x1EC1, 0x1EC1, 31, + 0x1EC3, 0x1EC3, 31, 0x1EC5, 0x1EC5, 31, 0x1EC7, 0x1EC7, 31, 0x1EC9, 0x1EC9, 31, + 0x1ECB, 0x1ECB, 31, 0x1ECD, 0x1ECD, 31, 0x1ECF, 0x1ECF, 31, 0x1ED1, 0x1ED1, 31, + 0x1ED3, 0x1ED3, 31, 0x1ED5, 0x1ED5, 31, 0x1ED7, 0x1ED7, 31, 0x1ED9, 0x1ED9, 31, + 0x1EDB, 0x1EDB, 31, 0x1EDD, 0x1EDD, 31, 0x1EDF, 0x1EDF, 31, 0x1EE1, 0x1EE1, 31, + 0x1EE3, 0x1EE3, 31, 0x1EE5, 0x1EE5, 31, 0x1EE7, 0x1EE7, 31, 0x1EE9, 0x1EE9, 31, + 0x1EEB, 0x1EEB, 31, 0x1EED, 0x1EED, 31, 0x1EEF, 0x1EEF, 31, 0x1EF1, 0x1EF1, 31, + 0x1EF3, 0x1EF3, 31, 0x1EF5, 0x1EF5, 31, 0x1EF7, 0x1EF7, 31, 0x1EF9, 0x1EF9, 31, + 0x1EFB, 0x1EFB, 31, 0x1EFD, 0x1EFD, 31, 0x1EFF, 0x1F07, 31, 0x1F10, 0x1F15, 31, + 0x1F20, 0x1F27, 31, 0x1F30, 0x1F37, 31, 0x1F40, 0x1F45, 31, 0x1F50, 0x1F57, 31, + 0x1F60, 0x1F67, 31, 0x1F70, 0x1F7D, 31, 0x1F80, 0x1F87, 31, 0x1F90, 0x1F97, 31, + 0x1FA0, 0x1FA7, 31, 0x1FB0, 0x1FB4, 31, 0x1FB6, 0x1FB7, 31, 0x1FBE, 0x1FBE, 31, + 0x1FC2, 0x1FC4, 31, 0x1FC6, 0x1FC7, 31, 0x1FD0, 0x1FD3, 31, 0x1FD6, 0x1FD7, 31, + 0x1FE0, 0x1FE7, 31, 0x1FF2, 0x1FF4, 31, 0x1FF6, 0x1FF7, 31, 0x210A, 0x210A, 31, + 0x210E, 0x210F, 31, 0x2113, 0x2113, 31, 0x212F, 0x212F, 31, 0x2134, 0x2134, 31, + 0x2139, 0x2139, 31, 0x213C, 0x213D, 31, 0x2146, 0x2149, 31, 0x214E, 0x214E, 31, + 0x2184, 0x2184, 31, 0x2C30, 0x2C5F, 31, 0x2C61, 0x2C61, 31, 0x2C65, 0x2C66, 31, + 0x2C68, 0x2C68, 31, 0x2C6A, 0x2C6A, 31, 0x2C6C, 0x2C6C, 31, 0x2C71, 0x2C71, 31, + }; + } + + private static int[] rangeChunk15() { + return new int[] { + 0x2C73, 0x2C74, 31, 0x2C76, 0x2C7B, 31, 0x2C81, 0x2C81, 31, 0x2C83, 0x2C83, 31, + 0x2C85, 0x2C85, 31, 0x2C87, 0x2C87, 31, 0x2C89, 0x2C89, 31, 0x2C8B, 0x2C8B, 31, + 0x2C8D, 0x2C8D, 31, 0x2C8F, 0x2C8F, 31, 0x2C91, 0x2C91, 31, 0x2C93, 0x2C93, 31, + 0x2C95, 0x2C95, 31, 0x2C97, 0x2C97, 31, 0x2C99, 0x2C99, 31, 0x2C9B, 0x2C9B, 31, + 0x2C9D, 0x2C9D, 31, 0x2C9F, 0x2C9F, 31, 0x2CA1, 0x2CA1, 31, 0x2CA3, 0x2CA3, 31, + 0x2CA5, 0x2CA5, 31, 0x2CA7, 0x2CA7, 31, 0x2CA9, 0x2CA9, 31, 0x2CAB, 0x2CAB, 31, + 0x2CAD, 0x2CAD, 31, 0x2CAF, 0x2CAF, 31, 0x2CB1, 0x2CB1, 31, 0x2CB3, 0x2CB3, 31, + 0x2CB5, 0x2CB5, 31, 0x2CB7, 0x2CB7, 31, 0x2CB9, 0x2CB9, 31, 0x2CBB, 0x2CBB, 31, + 0x2CBD, 0x2CBD, 31, 0x2CBF, 0x2CBF, 31, 0x2CC1, 0x2CC1, 31, 0x2CC3, 0x2CC3, 31, + 0x2CC5, 0x2CC5, 31, 0x2CC7, 0x2CC7, 31, 0x2CC9, 0x2CC9, 31, 0x2CCB, 0x2CCB, 31, + 0x2CCD, 0x2CCD, 31, 0x2CCF, 0x2CCF, 31, 0x2CD1, 0x2CD1, 31, 0x2CD3, 0x2CD3, 31, + 0x2CD5, 0x2CD5, 31, 0x2CD7, 0x2CD7, 31, 0x2CD9, 0x2CD9, 31, 0x2CDB, 0x2CDB, 31, + 0x2CDD, 0x2CDD, 31, 0x2CDF, 0x2CDF, 31, 0x2CE1, 0x2CE1, 31, 0x2CE3, 0x2CE4, 31, + 0x2CEC, 0x2CEC, 31, 0x2CEE, 0x2CEE, 31, 0x2CF3, 0x2CF3, 31, 0x2D00, 0x2D25, 31, + 0x2D27, 0x2D27, 31, 0x2D2D, 0x2D2D, 31, 0xA641, 0xA641, 31, 0xA643, 0xA643, 31, + 0xA645, 0xA645, 31, 0xA647, 0xA647, 31, 0xA649, 0xA649, 31, 0xA64B, 0xA64B, 31, + 0xA64D, 0xA64D, 31, 0xA64F, 0xA64F, 31, 0xA651, 0xA651, 31, 0xA653, 0xA653, 31, + 0xA655, 0xA655, 31, 0xA657, 0xA657, 31, 0xA659, 0xA659, 31, 0xA65B, 0xA65B, 31, + 0xA65D, 0xA65D, 31, 0xA65F, 0xA65F, 31, 0xA661, 0xA661, 31, 0xA663, 0xA663, 31, + 0xA665, 0xA665, 31, 0xA667, 0xA667, 31, 0xA669, 0xA669, 31, 0xA66B, 0xA66B, 31, + 0xA66D, 0xA66D, 31, 0xA681, 0xA681, 31, 0xA683, 0xA683, 31, 0xA685, 0xA685, 31, + 0xA687, 0xA687, 31, 0xA689, 0xA689, 31, 0xA68B, 0xA68B, 31, 0xA68D, 0xA68D, 31, + 0xA68F, 0xA68F, 31, 0xA691, 0xA691, 31, 0xA693, 0xA693, 31, 0xA695, 0xA695, 31, + 0xA697, 0xA697, 31, 0xA699, 0xA699, 31, 0xA69B, 0xA69B, 31, 0xA723, 0xA723, 31, + 0xA725, 0xA725, 31, 0xA727, 0xA727, 31, 0xA729, 0xA729, 31, 0xA72B, 0xA72B, 31, + 0xA72D, 0xA72D, 31, 0xA72F, 0xA731, 31, 0xA733, 0xA733, 31, 0xA735, 0xA735, 31, + 0xA737, 0xA737, 31, 0xA739, 0xA739, 31, 0xA73B, 0xA73B, 31, 0xA73D, 0xA73D, 31, + 0xA73F, 0xA73F, 31, 0xA741, 0xA741, 31, 0xA743, 0xA743, 31, 0xA745, 0xA745, 31, + 0xA747, 0xA747, 31, 0xA749, 0xA749, 31, 0xA74B, 0xA74B, 31, 0xA74D, 0xA74D, 31, + 0xA74F, 0xA74F, 31, 0xA751, 0xA751, 31, 0xA753, 0xA753, 31, 0xA755, 0xA755, 31, + 0xA757, 0xA757, 31, 0xA759, 0xA759, 31, 0xA75B, 0xA75B, 31, 0xA75D, 0xA75D, 31, + 0xA75F, 0xA75F, 31, 0xA761, 0xA761, 31, 0xA763, 0xA763, 31, 0xA765, 0xA765, 31, + 0xA767, 0xA767, 31, 0xA769, 0xA769, 31, 0xA76B, 0xA76B, 31, 0xA76D, 0xA76D, 31, + 0xA76F, 0xA76F, 31, 0xA771, 0xA778, 31, 0xA77A, 0xA77A, 31, 0xA77C, 0xA77C, 31, + 0xA77F, 0xA77F, 31, 0xA781, 0xA781, 31, 0xA783, 0xA783, 31, 0xA785, 0xA785, 31, + 0xA787, 0xA787, 31, 0xA78C, 0xA78C, 31, 0xA78E, 0xA78E, 31, 0xA791, 0xA791, 31, + 0xA793, 0xA795, 31, 0xA797, 0xA797, 31, 0xA799, 0xA799, 31, 0xA79B, 0xA79B, 31, + 0xA79D, 0xA79D, 31, 0xA79F, 0xA79F, 31, 0xA7A1, 0xA7A1, 31, 0xA7A3, 0xA7A3, 31, + 0xA7A5, 0xA7A5, 31, 0xA7A7, 0xA7A7, 31, 0xA7A9, 0xA7A9, 31, 0xA7AF, 0xA7AF, 31, + 0xA7B5, 0xA7B5, 31, 0xA7B7, 0xA7B7, 31, 0xA7B9, 0xA7B9, 31, 0xA7BB, 0xA7BB, 31, + 0xA7BD, 0xA7BD, 31, 0xA7BF, 0xA7BF, 31, 0xA7C1, 0xA7C1, 31, 0xA7C3, 0xA7C3, 31, + 0xA7C8, 0xA7C8, 31, 0xA7CA, 0xA7CA, 31, 0xA7CD, 0xA7CD, 31, 0xA7CF, 0xA7CF, 31, + 0xA7D1, 0xA7D1, 31, 0xA7D3, 0xA7D3, 31, 0xA7D5, 0xA7D5, 31, 0xA7D7, 0xA7D7, 31, + 0xA7D9, 0xA7D9, 31, 0xA7DB, 0xA7DB, 31, 0xA7F6, 0xA7F6, 31, 0xA7FA, 0xA7FA, 31, + 0xAB30, 0xAB5A, 31, 0xAB60, 0xAB68, 31, 0xAB70, 0xABBF, 31, 0xFB00, 0xFB06, 31, + 0xFB13, 0xFB17, 31, 0xFF41, 0xFF5A, 31, 0x10428, 0x1044F, 31, 0x104D8, 0x104FB, 31, + 0x10597, 0x105A1, 31, 0x105A3, 0x105B1, 31, 0x105B3, 0x105B9, 31, 0x105BB, 0x105BC, 31, + 0x10CC0, 0x10CF2, 31, 0x10D70, 0x10D85, 31, 0x118C0, 0x118DF, 31, 0x16E60, 0x16E7F, 31, + 0x16EBB, 0x16ED3, 31, 0x1D41A, 0x1D433, 31, 0x1D44E, 0x1D454, 31, 0x1D456, 0x1D467, 31, + 0x1D482, 0x1D49B, 31, 0x1D4B6, 0x1D4B9, 31, 0x1D4BB, 0x1D4BB, 31, 0x1D4BD, 0x1D4C3, 31, + 0x1D4C5, 0x1D4CF, 31, 0x1D4EA, 0x1D503, 31, 0x1D51E, 0x1D537, 31, 0x1D552, 0x1D56B, 31, + 0x1D586, 0x1D59F, 31, 0x1D5BA, 0x1D5D3, 31, 0x1D5EE, 0x1D607, 31, 0x1D622, 0x1D63B, 31, + 0x1D656, 0x1D66F, 31, 0x1D68A, 0x1D6A5, 31, 0x1D6C2, 0x1D6DA, 31, 0x1D6DC, 0x1D6E1, 31, + 0x1D6FC, 0x1D714, 31, 0x1D716, 0x1D71B, 31, 0x1D736, 0x1D74E, 31, 0x1D750, 0x1D755, 31, + 0x1D770, 0x1D788, 31, 0x1D78A, 0x1D78F, 31, 0x1D7AA, 0x1D7C2, 31, 0x1D7C4, 0x1D7C9, 31, + 0x1D7CB, 0x1D7CB, 31, 0x1DF00, 0x1DF09, 31, 0x1DF0B, 0x1DF1E, 31, 0x1DF25, 0x1DF2A, 31, + 0x1E922, 0x1E943, 31, 0x1C5, 0x1C5, 31, 0x1C8, 0x1C8, 31, 0x1CB, 0x1CB, 31, + 0x1F2, 0x1F2, 31, 0x1F88, 0x1F8F, 31, 0x1F98, 0x1F9F, 31, 0x1FA8, 0x1FAF, 31, + 0x1FBC, 0x1FBC, 31, 0x1FCC, 0x1FCC, 31, 0x1FFC, 0x1FFC, 31, 0x2B0, 0x2C1, 31, + 0x2C6, 0x2D1, 31, 0x2E0, 0x2E4, 31, 0x2EC, 0x2EC, 31, 0x2EE, 0x2EE, 31, + 0x374, 0x374, 31, 0x37A, 0x37A, 31, 0x559, 0x559, 31, 0x640, 0x640, 31, + 0x6E5, 0x6E6, 31, 0x7F4, 0x7F5, 31, 0x7FA, 0x7FA, 31, 0x81A, 0x81A, 31, + 0x824, 0x824, 31, 0x828, 0x828, 31, 0x8C9, 0x8C9, 31, 0x971, 0x971, 31, + 0xE46, 0xE46, 31, 0xEC6, 0xEC6, 31, 0x10FC, 0x10FC, 31, 0x17D7, 0x17D7, 31, + 0x1843, 0x1843, 31, 0x1AA7, 0x1AA7, 31, 0x1C78, 0x1C7D, 31, 0x1D2C, 0x1D6A, 31, + 0x1D78, 0x1D78, 31, 0x1D9B, 0x1DBF, 31, 0x2071, 0x2071, 31, 0x207F, 0x207F, 31, + 0x2090, 0x209C, 31, 0x2C7C, 0x2C7D, 31, 0x2D6F, 0x2D6F, 31, 0x2E2F, 0x2E2F, 31, + 0x3005, 0x3005, 31, 0x3031, 0x3035, 31, 0x303B, 0x303B, 31, 0x309D, 0x309E, 31, + 0x30FC, 0x30FE, 31, 0xA015, 0xA015, 31, 0xA4F8, 0xA4FD, 31, 0xA60C, 0xA60C, 31, + 0xA67F, 0xA67F, 31, 0xA69C, 0xA69D, 31, 0xA717, 0xA71F, 31, 0xA770, 0xA770, 31, + 0xA788, 0xA788, 31, 0xA7F1, 0xA7F4, 31, 0xA7F8, 0xA7F9, 31, 0xA9CF, 0xA9CF, 31, + 0xA9E6, 0xA9E6, 31, 0xAA70, 0xAA70, 31, 0xAADD, 0xAADD, 31, 0xAAF3, 0xAAF4, 31, + 0xAB5C, 0xAB5F, 31, 0xAB69, 0xAB69, 31, 0xFF70, 0xFF70, 31, 0xFF9E, 0xFF9F, 31, + 0x10780, 0x10785, 31, 0x10787, 0x107B0, 31, 0x107B2, 0x107BA, 31, 0x10D4E, 0x10D4E, 31, + 0x10D6F, 0x10D6F, 31, 0x10EC5, 0x10EC5, 31, 0x11DD9, 0x11DD9, 31, 0x16B40, 0x16B43, 31, + 0x16D40, 0x16D42, 31, 0x16D6B, 0x16D6C, 31, 0x16F93, 0x16F9F, 31, 0x16FE0, 0x16FE1, 31, + 0x16FE3, 0x16FE3, 31, 0x16FF2, 0x16FF3, 31, 0x1AFF0, 0x1AFF3, 31, 0x1AFF5, 0x1AFFB, 31, + 0x1AFFD, 0x1AFFE, 31, 0x1E030, 0x1E06D, 31, 0x1E137, 0x1E13D, 31, 0x1E4EB, 0x1E4EB, 31, + 0x1E6FF, 0x1E6FF, 31, 0x1E94B, 0x1E94B, 31, 0xAA, 0xAA, 31, 0xBA, 0xBA, 31, + 0x1BB, 0x1BB, 31, 0x1C0, 0x1C3, 31, 0x294, 0x295, 31, 0x5D0, 0x5EA, 31, + 0x5EF, 0x5F2, 31, 0x620, 0x63F, 31, 0x641, 0x64A, 31, 0x66E, 0x66F, 31, + 0x671, 0x6D3, 31, 0x6D5, 0x6D5, 31, 0x6EE, 0x6EF, 31, 0x6FA, 0x6FC, 31, + 0x6FF, 0x6FF, 31, 0x710, 0x710, 31, 0x712, 0x72F, 31, 0x74D, 0x7A5, 31, + 0x7B1, 0x7B1, 31, 0x7CA, 0x7EA, 31, 0x800, 0x815, 31, 0x840, 0x858, 31, + 0x860, 0x86A, 31, 0x870, 0x887, 31, 0x889, 0x88F, 31, 0x8A0, 0x8C8, 31, + 0x904, 0x939, 31, 0x93D, 0x93D, 31, 0x950, 0x950, 31, 0x958, 0x961, 31, + 0x972, 0x980, 31, 0x985, 0x98C, 31, 0x98F, 0x990, 31, 0x993, 0x9A8, 31, + 0x9AA, 0x9B0, 31, 0x9B2, 0x9B2, 31, 0x9B6, 0x9B9, 31, 0x9BD, 0x9BD, 31, + 0x9CE, 0x9CE, 31, 0x9DC, 0x9DD, 31, 0x9DF, 0x9E1, 31, 0x9F0, 0x9F1, 31, + 0x9FC, 0x9FC, 31, 0xA05, 0xA0A, 31, 0xA0F, 0xA10, 31, 0xA13, 0xA28, 31, + 0xA2A, 0xA30, 31, 0xA32, 0xA33, 31, 0xA35, 0xA36, 31, 0xA38, 0xA39, 31, + 0xA59, 0xA5C, 31, 0xA5E, 0xA5E, 31, 0xA72, 0xA74, 31, 0xA85, 0xA8D, 31, + 0xA8F, 0xA91, 31, 0xA93, 0xAA8, 31, 0xAAA, 0xAB0, 31, 0xAB2, 0xAB3, 31, + 0xAB5, 0xAB9, 31, 0xABD, 0xABD, 31, 0xAD0, 0xAD0, 31, 0xAE0, 0xAE1, 31, + 0xAF9, 0xAF9, 31, 0xB05, 0xB0C, 31, 0xB0F, 0xB10, 31, 0xB13, 0xB28, 31, + 0xB2A, 0xB30, 31, 0xB32, 0xB33, 31, 0xB35, 0xB39, 31, 0xB3D, 0xB3D, 31, + 0xB5C, 0xB5D, 31, 0xB5F, 0xB61, 31, 0xB71, 0xB71, 31, 0xB83, 0xB83, 31, + 0xB85, 0xB8A, 31, 0xB8E, 0xB90, 31, 0xB92, 0xB95, 31, 0xB99, 0xB9A, 31, + 0xB9C, 0xB9C, 31, 0xB9E, 0xB9F, 31, 0xBA3, 0xBA4, 31, 0xBA8, 0xBAA, 31, + 0xBAE, 0xBB9, 31, 0xBD0, 0xBD0, 31, 0xC05, 0xC0C, 31, 0xC0E, 0xC10, 31, + }; + } + + private static int[] rangeChunk16() { + return new int[] { + 0xC12, 0xC28, 31, 0xC2A, 0xC39, 31, 0xC3D, 0xC3D, 31, 0xC58, 0xC5A, 31, + 0xC5C, 0xC5D, 31, 0xC60, 0xC61, 31, 0xC80, 0xC80, 31, 0xC85, 0xC8C, 31, + 0xC8E, 0xC90, 31, 0xC92, 0xCA8, 31, 0xCAA, 0xCB3, 31, 0xCB5, 0xCB9, 31, + 0xCBD, 0xCBD, 31, 0xCDC, 0xCDE, 31, 0xCE0, 0xCE1, 31, 0xCF1, 0xCF2, 31, + 0xD04, 0xD0C, 31, 0xD0E, 0xD10, 31, 0xD12, 0xD3A, 31, 0xD3D, 0xD3D, 31, + 0xD4E, 0xD4E, 31, 0xD54, 0xD56, 31, 0xD5F, 0xD61, 31, 0xD7A, 0xD7F, 31, + 0xD85, 0xD96, 31, 0xD9A, 0xDB1, 31, 0xDB3, 0xDBB, 31, 0xDBD, 0xDBD, 31, + 0xDC0, 0xDC6, 31, 0xE01, 0xE30, 31, 0xE32, 0xE33, 31, 0xE40, 0xE45, 31, + 0xE81, 0xE82, 31, 0xE84, 0xE84, 31, 0xE86, 0xE8A, 31, 0xE8C, 0xEA3, 31, + 0xEA5, 0xEA5, 31, 0xEA7, 0xEB0, 31, 0xEB2, 0xEB3, 31, 0xEBD, 0xEBD, 31, + 0xEC0, 0xEC4, 31, 0xEDC, 0xEDF, 31, 0xF00, 0xF00, 31, 0xF40, 0xF47, 31, + 0xF49, 0xF6C, 31, 0xF88, 0xF8C, 31, 0x1000, 0x102A, 31, 0x103F, 0x103F, 31, + 0x1050, 0x1055, 31, 0x105A, 0x105D, 31, 0x1061, 0x1061, 31, 0x1065, 0x1066, 31, + 0x106E, 0x1070, 31, 0x1075, 0x1081, 31, 0x108E, 0x108E, 31, 0x1100, 0x1248, 31, + 0x124A, 0x124D, 31, 0x1250, 0x1256, 31, 0x1258, 0x1258, 31, 0x125A, 0x125D, 31, + 0x1260, 0x1288, 31, 0x128A, 0x128D, 31, 0x1290, 0x12B0, 31, 0x12B2, 0x12B5, 31, + 0x12B8, 0x12BE, 31, 0x12C0, 0x12C0, 31, 0x12C2, 0x12C5, 31, 0x12C8, 0x12D6, 31, + 0x12D8, 0x1310, 31, 0x1312, 0x1315, 31, 0x1318, 0x135A, 31, 0x1380, 0x138F, 31, + 0x1401, 0x166C, 31, 0x166F, 0x167F, 31, 0x1681, 0x169A, 31, 0x16A0, 0x16EA, 31, + 0x16F1, 0x16F8, 31, 0x1700, 0x1711, 31, 0x171F, 0x1731, 31, 0x1740, 0x1751, 31, + 0x1760, 0x176C, 31, 0x176E, 0x1770, 31, 0x1780, 0x17B3, 31, 0x17DC, 0x17DC, 31, + 0x1820, 0x1842, 31, 0x1844, 0x1878, 31, 0x1880, 0x1884, 31, 0x1887, 0x18A8, 31, + 0x18AA, 0x18AA, 31, 0x18B0, 0x18F5, 31, 0x1900, 0x191E, 31, 0x1950, 0x196D, 31, + 0x1970, 0x1974, 31, 0x1980, 0x19AB, 31, 0x19B0, 0x19C9, 31, 0x1A00, 0x1A16, 31, + 0x1A20, 0x1A54, 31, 0x1B05, 0x1B33, 31, 0x1B45, 0x1B4C, 31, 0x1B83, 0x1BA0, 31, + 0x1BAE, 0x1BAF, 31, 0x1BBA, 0x1BE5, 31, 0x1C00, 0x1C23, 31, 0x1C4D, 0x1C4F, 31, + 0x1C5A, 0x1C77, 31, 0x1CE9, 0x1CEC, 31, 0x1CEE, 0x1CF3, 31, 0x1CF5, 0x1CF6, 31, + 0x1CFA, 0x1CFA, 31, 0x2135, 0x2138, 31, 0x2D30, 0x2D67, 31, 0x2D80, 0x2D96, 31, + 0x2DA0, 0x2DA6, 31, 0x2DA8, 0x2DAE, 31, 0x2DB0, 0x2DB6, 31, 0x2DB8, 0x2DBE, 31, + 0x2DC0, 0x2DC6, 31, 0x2DC8, 0x2DCE, 31, 0x2DD0, 0x2DD6, 31, 0x2DD8, 0x2DDE, 31, + 0x3006, 0x3006, 31, 0x303C, 0x303C, 31, 0x3041, 0x3096, 31, 0x309F, 0x309F, 31, + 0x30A1, 0x30FA, 31, 0x30FF, 0x30FF, 31, 0x3105, 0x312F, 31, 0x3131, 0x318E, 31, + 0x31A0, 0x31BF, 31, 0x31F0, 0x31FF, 31, 0x3400, 0x4DBF, 31, 0x4E00, 0xA014, 31, + 0xA016, 0xA48C, 31, 0xA4D0, 0xA4F7, 31, 0xA500, 0xA60B, 31, 0xA610, 0xA61F, 31, + 0xA62A, 0xA62B, 31, 0xA66E, 0xA66E, 31, 0xA6A0, 0xA6E5, 31, 0xA78F, 0xA78F, 31, + 0xA7F7, 0xA7F7, 31, 0xA7FB, 0xA801, 31, 0xA803, 0xA805, 31, 0xA807, 0xA80A, 31, + 0xA80C, 0xA822, 31, 0xA840, 0xA873, 31, 0xA882, 0xA8B3, 31, 0xA8F2, 0xA8F7, 31, + 0xA8FB, 0xA8FB, 31, 0xA8FD, 0xA8FE, 31, 0xA90A, 0xA925, 31, 0xA930, 0xA946, 31, + 0xA960, 0xA97C, 31, 0xA984, 0xA9B2, 31, 0xA9E0, 0xA9E4, 31, 0xA9E7, 0xA9EF, 31, + 0xA9FA, 0xA9FE, 31, 0xAA00, 0xAA28, 31, 0xAA40, 0xAA42, 31, 0xAA44, 0xAA4B, 31, + 0xAA60, 0xAA6F, 31, 0xAA71, 0xAA76, 31, 0xAA7A, 0xAA7A, 31, 0xAA7E, 0xAAAF, 31, + 0xAAB1, 0xAAB1, 31, 0xAAB5, 0xAAB6, 31, 0xAAB9, 0xAABD, 31, 0xAAC0, 0xAAC0, 31, + 0xAAC2, 0xAAC2, 31, 0xAADB, 0xAADC, 31, 0xAAE0, 0xAAEA, 31, 0xAAF2, 0xAAF2, 31, + 0xAB01, 0xAB06, 31, 0xAB09, 0xAB0E, 31, 0xAB11, 0xAB16, 31, 0xAB20, 0xAB26, 31, + 0xAB28, 0xAB2E, 31, 0xABC0, 0xABE2, 31, 0xAC00, 0xD7A3, 31, 0xD7B0, 0xD7C6, 31, + 0xD7CB, 0xD7FB, 31, 0xF900, 0xFA6D, 31, 0xFA70, 0xFAD9, 31, 0xFB1D, 0xFB1D, 31, + 0xFB1F, 0xFB28, 31, 0xFB2A, 0xFB36, 31, 0xFB38, 0xFB3C, 31, 0xFB3E, 0xFB3E, 31, + 0xFB40, 0xFB41, 31, 0xFB43, 0xFB44, 31, 0xFB46, 0xFBB1, 31, 0xFBD3, 0xFD3D, 31, + 0xFD50, 0xFD8F, 31, 0xFD92, 0xFDC7, 31, 0xFDF0, 0xFDFB, 31, 0xFE70, 0xFE74, 31, + 0xFE76, 0xFEFC, 31, 0xFF66, 0xFF6F, 31, 0xFF71, 0xFF9D, 31, 0xFFA0, 0xFFBE, 31, + 0xFFC2, 0xFFC7, 31, 0xFFCA, 0xFFCF, 31, 0xFFD2, 0xFFD7, 31, 0xFFDA, 0xFFDC, 31, + 0x10000, 0x1000B, 31, 0x1000D, 0x10026, 31, 0x10028, 0x1003A, 31, 0x1003C, 0x1003D, 31, + 0x1003F, 0x1004D, 31, 0x10050, 0x1005D, 31, 0x10080, 0x100FA, 31, 0x10280, 0x1029C, 31, + 0x102A0, 0x102D0, 31, 0x10300, 0x1031F, 31, 0x1032D, 0x10340, 31, 0x10342, 0x10349, 31, + 0x10350, 0x10375, 31, 0x10380, 0x1039D, 31, 0x103A0, 0x103C3, 31, 0x103C8, 0x103CF, 31, + 0x10450, 0x1049D, 31, 0x10500, 0x10527, 31, 0x10530, 0x10563, 31, 0x105C0, 0x105F3, 31, + 0x10600, 0x10736, 31, 0x10740, 0x10755, 31, 0x10760, 0x10767, 31, 0x10800, 0x10805, 31, + 0x10808, 0x10808, 31, 0x1080A, 0x10835, 31, 0x10837, 0x10838, 31, 0x1083C, 0x1083C, 31, + 0x1083F, 0x10855, 31, 0x10860, 0x10876, 31, 0x10880, 0x1089E, 31, 0x108E0, 0x108F2, 31, + 0x108F4, 0x108F5, 31, 0x10900, 0x10915, 31, 0x10920, 0x10939, 31, 0x10940, 0x10959, 31, + 0x10980, 0x109B7, 31, 0x109BE, 0x109BF, 31, 0x10A00, 0x10A00, 31, 0x10A10, 0x10A13, 31, + 0x10A15, 0x10A17, 31, 0x10A19, 0x10A35, 31, 0x10A60, 0x10A7C, 31, 0x10A80, 0x10A9C, 31, + 0x10AC0, 0x10AC7, 31, 0x10AC9, 0x10AE4, 31, 0x10B00, 0x10B35, 31, 0x10B40, 0x10B55, 31, + 0x10B60, 0x10B72, 31, 0x10B80, 0x10B91, 31, 0x10C00, 0x10C48, 31, 0x10D00, 0x10D23, 31, + 0x10D4A, 0x10D4D, 31, 0x10D4F, 0x10D4F, 31, 0x10E80, 0x10EA9, 31, 0x10EB0, 0x10EB1, 31, + 0x10EC2, 0x10EC4, 31, 0x10EC6, 0x10EC7, 31, 0x10F00, 0x10F1C, 31, 0x10F27, 0x10F27, 31, + 0x10F30, 0x10F45, 31, 0x10F70, 0x10F81, 31, 0x10FB0, 0x10FC4, 31, 0x10FE0, 0x10FF6, 31, + 0x11003, 0x11037, 31, 0x11071, 0x11072, 31, 0x11075, 0x11075, 31, 0x11083, 0x110AF, 31, + 0x110D0, 0x110E8, 31, 0x11103, 0x11126, 31, 0x11144, 0x11144, 31, 0x11147, 0x11147, 31, + 0x11150, 0x11172, 31, 0x11176, 0x11176, 31, 0x11183, 0x111B2, 31, 0x111C1, 0x111C4, 31, + 0x111DA, 0x111DA, 31, 0x111DC, 0x111DC, 31, 0x11200, 0x11211, 31, 0x11213, 0x1122B, 31, + 0x1123F, 0x11240, 31, 0x11280, 0x11286, 31, 0x11288, 0x11288, 31, 0x1128A, 0x1128D, 31, + 0x1128F, 0x1129D, 31, 0x1129F, 0x112A8, 31, 0x112B0, 0x112DE, 31, 0x11305, 0x1130C, 31, + 0x1130F, 0x11310, 31, 0x11313, 0x11328, 31, 0x1132A, 0x11330, 31, 0x11332, 0x11333, 31, + 0x11335, 0x11339, 31, 0x1133D, 0x1133D, 31, 0x11350, 0x11350, 31, 0x1135D, 0x11361, 31, + 0x11380, 0x11389, 31, 0x1138B, 0x1138B, 31, 0x1138E, 0x1138E, 31, 0x11390, 0x113B5, 31, + 0x113B7, 0x113B7, 31, 0x113D1, 0x113D1, 31, 0x113D3, 0x113D3, 31, 0x11400, 0x11434, 31, + 0x11447, 0x1144A, 31, 0x1145F, 0x11461, 31, 0x11480, 0x114AF, 31, 0x114C4, 0x114C5, 31, + 0x114C7, 0x114C7, 31, 0x11580, 0x115AE, 31, 0x115D8, 0x115DB, 31, 0x11600, 0x1162F, 31, + 0x11644, 0x11644, 31, 0x11680, 0x116AA, 31, 0x116B8, 0x116B8, 31, 0x11700, 0x1171A, 31, + 0x11740, 0x11746, 31, 0x11800, 0x1182B, 31, 0x118FF, 0x11906, 31, 0x11909, 0x11909, 31, + 0x1190C, 0x11913, 31, 0x11915, 0x11916, 31, 0x11918, 0x1192F, 31, 0x1193F, 0x1193F, 31, + 0x11941, 0x11941, 31, 0x119A0, 0x119A7, 31, 0x119AA, 0x119D0, 31, 0x119E1, 0x119E1, 31, + 0x119E3, 0x119E3, 31, 0x11A00, 0x11A00, 31, 0x11A0B, 0x11A32, 31, 0x11A3A, 0x11A3A, 31, + 0x11A50, 0x11A50, 31, 0x11A5C, 0x11A89, 31, 0x11A9D, 0x11A9D, 31, 0x11AB0, 0x11AF8, 31, + 0x11BC0, 0x11BE0, 31, 0x11C00, 0x11C08, 31, 0x11C0A, 0x11C2E, 31, 0x11C40, 0x11C40, 31, + 0x11C72, 0x11C8F, 31, 0x11D00, 0x11D06, 31, 0x11D08, 0x11D09, 31, 0x11D0B, 0x11D30, 31, + 0x11D46, 0x11D46, 31, 0x11D60, 0x11D65, 31, 0x11D67, 0x11D68, 31, 0x11D6A, 0x11D89, 31, + 0x11D98, 0x11D98, 31, 0x11DB0, 0x11DD8, 31, 0x11DDA, 0x11DDB, 31, 0x11EE0, 0x11EF2, 31, + 0x11F02, 0x11F02, 31, 0x11F04, 0x11F10, 31, 0x11F12, 0x11F33, 31, 0x11FB0, 0x11FB0, 31, + 0x12000, 0x12399, 31, 0x12480, 0x12543, 31, 0x12F90, 0x12FF0, 31, 0x13000, 0x1342F, 31, + 0x13441, 0x13446, 31, 0x13460, 0x143FA, 31, 0x14400, 0x14646, 31, 0x16100, 0x1611D, 31, + 0x16800, 0x16A38, 31, 0x16A40, 0x16A5E, 31, 0x16A70, 0x16ABE, 31, 0x16AD0, 0x16AED, 31, + 0x16B00, 0x16B2F, 31, 0x16B63, 0x16B77, 31, 0x16B7D, 0x16B8F, 31, 0x16D43, 0x16D6A, 31, + 0x16F00, 0x16F4A, 31, 0x16F50, 0x16F50, 31, 0x17000, 0x18CD5, 31, 0x18CFF, 0x18D1E, 31, + 0x18D80, 0x18DF2, 31, 0x1B000, 0x1B122, 31, 0x1B132, 0x1B132, 31, 0x1B150, 0x1B152, 31, + 0x1B155, 0x1B155, 31, 0x1B164, 0x1B167, 31, 0x1B170, 0x1B2FB, 31, 0x1BC00, 0x1BC6A, 31, + 0x1BC70, 0x1BC7C, 31, 0x1BC80, 0x1BC88, 31, 0x1BC90, 0x1BC99, 31, 0x1DF0A, 0x1DF0A, 31, + 0x1E100, 0x1E12C, 31, 0x1E14E, 0x1E14E, 31, 0x1E290, 0x1E2AD, 31, 0x1E2C0, 0x1E2EB, 31, + 0x1E4D0, 0x1E4EA, 31, 0x1E5D0, 0x1E5ED, 31, 0x1E5F0, 0x1E5F0, 31, 0x1E6C0, 0x1E6DE, 31, + }; + } + + private static int[] rangeChunk17() { + return new int[] { + 0x1E6E0, 0x1E6E2, 31, 0x1E6E4, 0x1E6E5, 31, 0x1E6E7, 0x1E6ED, 31, 0x1E6F0, 0x1E6F4, 31, + 0x1E6FE, 0x1E6FE, 31, 0x1E7E0, 0x1E7E6, 31, 0x1E7E8, 0x1E7EB, 31, 0x1E7ED, 0x1E7EE, 31, + 0x1E7F0, 0x1E7FE, 31, 0x1E800, 0x1E8C4, 31, 0x1EE00, 0x1EE03, 31, 0x1EE05, 0x1EE1F, 31, + 0x1EE21, 0x1EE22, 31, 0x1EE24, 0x1EE24, 31, 0x1EE27, 0x1EE27, 31, 0x1EE29, 0x1EE32, 31, + 0x1EE34, 0x1EE37, 31, 0x1EE39, 0x1EE39, 31, 0x1EE3B, 0x1EE3B, 31, 0x1EE42, 0x1EE42, 31, + 0x1EE47, 0x1EE47, 31, 0x1EE49, 0x1EE49, 31, 0x1EE4B, 0x1EE4B, 31, 0x1EE4D, 0x1EE4F, 31, + 0x1EE51, 0x1EE52, 31, 0x1EE54, 0x1EE54, 31, 0x1EE57, 0x1EE57, 31, 0x1EE59, 0x1EE59, 31, + 0x1EE5B, 0x1EE5B, 31, 0x1EE5D, 0x1EE5D, 31, 0x1EE5F, 0x1EE5F, 31, 0x1EE61, 0x1EE62, 31, + 0x1EE64, 0x1EE64, 31, 0x1EE67, 0x1EE6A, 31, 0x1EE6C, 0x1EE72, 31, 0x1EE74, 0x1EE77, 31, + 0x1EE79, 0x1EE7C, 31, 0x1EE7E, 0x1EE7E, 31, 0x1EE80, 0x1EE89, 31, 0x1EE8B, 0x1EE9B, 31, + 0x1EEA1, 0x1EEA3, 31, 0x1EEA5, 0x1EEA9, 31, 0x1EEAB, 0x1EEBB, 31, 0x20000, 0x2A6DF, 31, + 0x2A700, 0x2B81D, 31, 0x2B820, 0x2CEAD, 31, 0x2CEB0, 0x2EBE0, 31, 0x2EBF0, 0x2EE5D, 31, + 0x2F800, 0x2FA1D, 31, 0x30000, 0x3134A, 31, 0x31350, 0x33479, 31, 0x41, 0x5A, 32, + 0xC0, 0xD6, 32, 0xD8, 0xDE, 32, 0x100, 0x100, 32, 0x102, 0x102, 32, + 0x104, 0x104, 32, 0x106, 0x106, 32, 0x108, 0x108, 32, 0x10A, 0x10A, 32, + 0x10C, 0x10C, 32, 0x10E, 0x10E, 32, 0x110, 0x110, 32, 0x112, 0x112, 32, + 0x114, 0x114, 32, 0x116, 0x116, 32, 0x118, 0x118, 32, 0x11A, 0x11A, 32, + 0x11C, 0x11C, 32, 0x11E, 0x11E, 32, 0x120, 0x120, 32, 0x122, 0x122, 32, + 0x124, 0x124, 32, 0x126, 0x126, 32, 0x128, 0x128, 32, 0x12A, 0x12A, 32, + 0x12C, 0x12C, 32, 0x12E, 0x12E, 32, 0x130, 0x130, 32, 0x132, 0x132, 32, + 0x134, 0x134, 32, 0x136, 0x136, 32, 0x139, 0x139, 32, 0x13B, 0x13B, 32, + 0x13D, 0x13D, 32, 0x13F, 0x13F, 32, 0x141, 0x141, 32, 0x143, 0x143, 32, + 0x145, 0x145, 32, 0x147, 0x147, 32, 0x14A, 0x14A, 32, 0x14C, 0x14C, 32, + 0x14E, 0x14E, 32, 0x150, 0x150, 32, 0x152, 0x152, 32, 0x154, 0x154, 32, + 0x156, 0x156, 32, 0x158, 0x158, 32, 0x15A, 0x15A, 32, 0x15C, 0x15C, 32, + 0x15E, 0x15E, 32, 0x160, 0x160, 32, 0x162, 0x162, 32, 0x164, 0x164, 32, + 0x166, 0x166, 32, 0x168, 0x168, 32, 0x16A, 0x16A, 32, 0x16C, 0x16C, 32, + 0x16E, 0x16E, 32, 0x170, 0x170, 32, 0x172, 0x172, 32, 0x174, 0x174, 32, + 0x176, 0x176, 32, 0x178, 0x179, 32, 0x17B, 0x17B, 32, 0x17D, 0x17D, 32, + 0x181, 0x182, 32, 0x184, 0x184, 32, 0x186, 0x187, 32, 0x189, 0x18B, 32, + 0x18E, 0x191, 32, 0x193, 0x194, 32, 0x196, 0x198, 32, 0x19C, 0x19D, 32, + 0x19F, 0x1A0, 32, 0x1A2, 0x1A2, 32, 0x1A4, 0x1A4, 32, 0x1A6, 0x1A7, 32, + 0x1A9, 0x1A9, 32, 0x1AC, 0x1AC, 32, 0x1AE, 0x1AF, 32, 0x1B1, 0x1B3, 32, + 0x1B5, 0x1B5, 32, 0x1B7, 0x1B8, 32, 0x1BC, 0x1BC, 32, 0x1C4, 0x1C4, 32, + 0x1C7, 0x1C7, 32, 0x1CA, 0x1CA, 32, 0x1CD, 0x1CD, 32, 0x1CF, 0x1CF, 32, + 0x1D1, 0x1D1, 32, 0x1D3, 0x1D3, 32, 0x1D5, 0x1D5, 32, 0x1D7, 0x1D7, 32, + 0x1D9, 0x1D9, 32, 0x1DB, 0x1DB, 32, 0x1DE, 0x1DE, 32, 0x1E0, 0x1E0, 32, + 0x1E2, 0x1E2, 32, 0x1E4, 0x1E4, 32, 0x1E6, 0x1E6, 32, 0x1E8, 0x1E8, 32, + 0x1EA, 0x1EA, 32, 0x1EC, 0x1EC, 32, 0x1EE, 0x1EE, 32, 0x1F1, 0x1F1, 32, + 0x1F4, 0x1F4, 32, 0x1F6, 0x1F8, 32, 0x1FA, 0x1FA, 32, 0x1FC, 0x1FC, 32, + 0x1FE, 0x1FE, 32, 0x200, 0x200, 32, 0x202, 0x202, 32, 0x204, 0x204, 32, + 0x206, 0x206, 32, 0x208, 0x208, 32, 0x20A, 0x20A, 32, 0x20C, 0x20C, 32, + 0x20E, 0x20E, 32, 0x210, 0x210, 32, 0x212, 0x212, 32, 0x214, 0x214, 32, + 0x216, 0x216, 32, 0x218, 0x218, 32, 0x21A, 0x21A, 32, 0x21C, 0x21C, 32, + 0x21E, 0x21E, 32, 0x220, 0x220, 32, 0x222, 0x222, 32, 0x224, 0x224, 32, + 0x226, 0x226, 32, 0x228, 0x228, 32, 0x22A, 0x22A, 32, 0x22C, 0x22C, 32, + 0x22E, 0x22E, 32, 0x230, 0x230, 32, 0x232, 0x232, 32, 0x23A, 0x23B, 32, + 0x23D, 0x23E, 32, 0x241, 0x241, 32, 0x243, 0x246, 32, 0x248, 0x248, 32, + 0x24A, 0x24A, 32, 0x24C, 0x24C, 32, 0x24E, 0x24E, 32, 0x370, 0x370, 32, + 0x372, 0x372, 32, 0x376, 0x376, 32, 0x37F, 0x37F, 32, 0x386, 0x386, 32, + 0x388, 0x38A, 32, 0x38C, 0x38C, 32, 0x38E, 0x38F, 32, 0x391, 0x3A1, 32, + 0x3A3, 0x3AB, 32, 0x3CF, 0x3CF, 32, 0x3D2, 0x3D4, 32, 0x3D8, 0x3D8, 32, + 0x3DA, 0x3DA, 32, 0x3DC, 0x3DC, 32, 0x3DE, 0x3DE, 32, 0x3E0, 0x3E0, 32, + 0x3E2, 0x3E2, 32, 0x3E4, 0x3E4, 32, 0x3E6, 0x3E6, 32, 0x3E8, 0x3E8, 32, + 0x3EA, 0x3EA, 32, 0x3EC, 0x3EC, 32, 0x3EE, 0x3EE, 32, 0x3F4, 0x3F4, 32, + 0x3F7, 0x3F7, 32, 0x3F9, 0x3FA, 32, 0x3FD, 0x42F, 32, 0x460, 0x460, 32, + 0x462, 0x462, 32, 0x464, 0x464, 32, 0x466, 0x466, 32, 0x468, 0x468, 32, + 0x46A, 0x46A, 32, 0x46C, 0x46C, 32, 0x46E, 0x46E, 32, 0x470, 0x470, 32, + 0x472, 0x472, 32, 0x474, 0x474, 32, 0x476, 0x476, 32, 0x478, 0x478, 32, + 0x47A, 0x47A, 32, 0x47C, 0x47C, 32, 0x47E, 0x47E, 32, 0x480, 0x480, 32, + 0x48A, 0x48A, 32, 0x48C, 0x48C, 32, 0x48E, 0x48E, 32, 0x490, 0x490, 32, + 0x492, 0x492, 32, 0x494, 0x494, 32, 0x496, 0x496, 32, 0x498, 0x498, 32, + 0x49A, 0x49A, 32, 0x49C, 0x49C, 32, 0x49E, 0x49E, 32, 0x4A0, 0x4A0, 32, + 0x4A2, 0x4A2, 32, 0x4A4, 0x4A4, 32, 0x4A6, 0x4A6, 32, 0x4A8, 0x4A8, 32, + 0x4AA, 0x4AA, 32, 0x4AC, 0x4AC, 32, 0x4AE, 0x4AE, 32, 0x4B0, 0x4B0, 32, + 0x4B2, 0x4B2, 32, 0x4B4, 0x4B4, 32, 0x4B6, 0x4B6, 32, 0x4B8, 0x4B8, 32, + 0x4BA, 0x4BA, 32, 0x4BC, 0x4BC, 32, 0x4BE, 0x4BE, 32, 0x4C0, 0x4C1, 32, + 0x4C3, 0x4C3, 32, 0x4C5, 0x4C5, 32, 0x4C7, 0x4C7, 32, 0x4C9, 0x4C9, 32, + 0x4CB, 0x4CB, 32, 0x4CD, 0x4CD, 32, 0x4D0, 0x4D0, 32, 0x4D2, 0x4D2, 32, + 0x4D4, 0x4D4, 32, 0x4D6, 0x4D6, 32, 0x4D8, 0x4D8, 32, 0x4DA, 0x4DA, 32, + 0x4DC, 0x4DC, 32, 0x4DE, 0x4DE, 32, 0x4E0, 0x4E0, 32, 0x4E2, 0x4E2, 32, + 0x4E4, 0x4E4, 32, 0x4E6, 0x4E6, 32, 0x4E8, 0x4E8, 32, 0x4EA, 0x4EA, 32, + 0x4EC, 0x4EC, 32, 0x4EE, 0x4EE, 32, 0x4F0, 0x4F0, 32, 0x4F2, 0x4F2, 32, + 0x4F4, 0x4F4, 32, 0x4F6, 0x4F6, 32, 0x4F8, 0x4F8, 32, 0x4FA, 0x4FA, 32, + 0x4FC, 0x4FC, 32, 0x4FE, 0x4FE, 32, 0x500, 0x500, 32, 0x502, 0x502, 32, + 0x504, 0x504, 32, 0x506, 0x506, 32, 0x508, 0x508, 32, 0x50A, 0x50A, 32, + 0x50C, 0x50C, 32, 0x50E, 0x50E, 32, 0x510, 0x510, 32, 0x512, 0x512, 32, + 0x514, 0x514, 32, 0x516, 0x516, 32, 0x518, 0x518, 32, 0x51A, 0x51A, 32, + 0x51C, 0x51C, 32, 0x51E, 0x51E, 32, 0x520, 0x520, 32, 0x522, 0x522, 32, + 0x524, 0x524, 32, 0x526, 0x526, 32, 0x528, 0x528, 32, 0x52A, 0x52A, 32, + 0x52C, 0x52C, 32, 0x52E, 0x52E, 32, 0x531, 0x556, 32, 0x10A0, 0x10C5, 32, + 0x10C7, 0x10C7, 32, 0x10CD, 0x10CD, 32, 0x13A0, 0x13F5, 32, 0x1C89, 0x1C89, 32, + 0x1C90, 0x1CBA, 32, 0x1CBD, 0x1CBF, 32, 0x1E00, 0x1E00, 32, 0x1E02, 0x1E02, 32, + 0x1E04, 0x1E04, 32, 0x1E06, 0x1E06, 32, 0x1E08, 0x1E08, 32, 0x1E0A, 0x1E0A, 32, + 0x1E0C, 0x1E0C, 32, 0x1E0E, 0x1E0E, 32, 0x1E10, 0x1E10, 32, 0x1E12, 0x1E12, 32, + 0x1E14, 0x1E14, 32, 0x1E16, 0x1E16, 32, 0x1E18, 0x1E18, 32, 0x1E1A, 0x1E1A, 32, + 0x1E1C, 0x1E1C, 32, 0x1E1E, 0x1E1E, 32, 0x1E20, 0x1E20, 32, 0x1E22, 0x1E22, 32, + 0x1E24, 0x1E24, 32, 0x1E26, 0x1E26, 32, 0x1E28, 0x1E28, 32, 0x1E2A, 0x1E2A, 32, + 0x1E2C, 0x1E2C, 32, 0x1E2E, 0x1E2E, 32, 0x1E30, 0x1E30, 32, 0x1E32, 0x1E32, 32, + 0x1E34, 0x1E34, 32, 0x1E36, 0x1E36, 32, 0x1E38, 0x1E38, 32, 0x1E3A, 0x1E3A, 32, + 0x1E3C, 0x1E3C, 32, 0x1E3E, 0x1E3E, 32, 0x1E40, 0x1E40, 32, 0x1E42, 0x1E42, 32, + 0x1E44, 0x1E44, 32, 0x1E46, 0x1E46, 32, 0x1E48, 0x1E48, 32, 0x1E4A, 0x1E4A, 32, + 0x1E4C, 0x1E4C, 32, 0x1E4E, 0x1E4E, 32, 0x1E50, 0x1E50, 32, 0x1E52, 0x1E52, 32, + 0x1E54, 0x1E54, 32, 0x1E56, 0x1E56, 32, 0x1E58, 0x1E58, 32, 0x1E5A, 0x1E5A, 32, + 0x1E5C, 0x1E5C, 32, 0x1E5E, 0x1E5E, 32, 0x1E60, 0x1E60, 32, 0x1E62, 0x1E62, 32, + 0x1E64, 0x1E64, 32, 0x1E66, 0x1E66, 32, 0x1E68, 0x1E68, 32, 0x1E6A, 0x1E6A, 32, + 0x1E6C, 0x1E6C, 32, 0x1E6E, 0x1E6E, 32, 0x1E70, 0x1E70, 32, 0x1E72, 0x1E72, 32, + 0x1E74, 0x1E74, 32, 0x1E76, 0x1E76, 32, 0x1E78, 0x1E78, 32, 0x1E7A, 0x1E7A, 32, + 0x1E7C, 0x1E7C, 32, 0x1E7E, 0x1E7E, 32, 0x1E80, 0x1E80, 32, 0x1E82, 0x1E82, 32, + 0x1E84, 0x1E84, 32, 0x1E86, 0x1E86, 32, 0x1E88, 0x1E88, 32, 0x1E8A, 0x1E8A, 32, + }; + } + + private static int[] rangeChunk18() { + return new int[] { + 0x1E8C, 0x1E8C, 32, 0x1E8E, 0x1E8E, 32, 0x1E90, 0x1E90, 32, 0x1E92, 0x1E92, 32, + 0x1E94, 0x1E94, 32, 0x1E9E, 0x1E9E, 32, 0x1EA0, 0x1EA0, 32, 0x1EA2, 0x1EA2, 32, + 0x1EA4, 0x1EA4, 32, 0x1EA6, 0x1EA6, 32, 0x1EA8, 0x1EA8, 32, 0x1EAA, 0x1EAA, 32, + 0x1EAC, 0x1EAC, 32, 0x1EAE, 0x1EAE, 32, 0x1EB0, 0x1EB0, 32, 0x1EB2, 0x1EB2, 32, + 0x1EB4, 0x1EB4, 32, 0x1EB6, 0x1EB6, 32, 0x1EB8, 0x1EB8, 32, 0x1EBA, 0x1EBA, 32, + 0x1EBC, 0x1EBC, 32, 0x1EBE, 0x1EBE, 32, 0x1EC0, 0x1EC0, 32, 0x1EC2, 0x1EC2, 32, + 0x1EC4, 0x1EC4, 32, 0x1EC6, 0x1EC6, 32, 0x1EC8, 0x1EC8, 32, 0x1ECA, 0x1ECA, 32, + 0x1ECC, 0x1ECC, 32, 0x1ECE, 0x1ECE, 32, 0x1ED0, 0x1ED0, 32, 0x1ED2, 0x1ED2, 32, + 0x1ED4, 0x1ED4, 32, 0x1ED6, 0x1ED6, 32, 0x1ED8, 0x1ED8, 32, 0x1EDA, 0x1EDA, 32, + 0x1EDC, 0x1EDC, 32, 0x1EDE, 0x1EDE, 32, 0x1EE0, 0x1EE0, 32, 0x1EE2, 0x1EE2, 32, + 0x1EE4, 0x1EE4, 32, 0x1EE6, 0x1EE6, 32, 0x1EE8, 0x1EE8, 32, 0x1EEA, 0x1EEA, 32, + 0x1EEC, 0x1EEC, 32, 0x1EEE, 0x1EEE, 32, 0x1EF0, 0x1EF0, 32, 0x1EF2, 0x1EF2, 32, + 0x1EF4, 0x1EF4, 32, 0x1EF6, 0x1EF6, 32, 0x1EF8, 0x1EF8, 32, 0x1EFA, 0x1EFA, 32, + 0x1EFC, 0x1EFC, 32, 0x1EFE, 0x1EFE, 32, 0x1F08, 0x1F0F, 32, 0x1F18, 0x1F1D, 32, + 0x1F28, 0x1F2F, 32, 0x1F38, 0x1F3F, 32, 0x1F48, 0x1F4D, 32, 0x1F59, 0x1F59, 32, + 0x1F5B, 0x1F5B, 32, 0x1F5D, 0x1F5D, 32, 0x1F5F, 0x1F5F, 32, 0x1F68, 0x1F6F, 32, + 0x1FB8, 0x1FBB, 32, 0x1FC8, 0x1FCB, 32, 0x1FD8, 0x1FDB, 32, 0x1FE8, 0x1FEC, 32, + 0x1FF8, 0x1FFB, 32, 0x2102, 0x2102, 32, 0x2107, 0x2107, 32, 0x210B, 0x210D, 32, + 0x2110, 0x2112, 32, 0x2115, 0x2115, 32, 0x2119, 0x211D, 32, 0x2124, 0x2124, 32, + 0x2126, 0x2126, 32, 0x2128, 0x2128, 32, 0x212A, 0x212D, 32, 0x2130, 0x2133, 32, + 0x213E, 0x213F, 32, 0x2145, 0x2145, 32, 0x2183, 0x2183, 32, 0x2C00, 0x2C2F, 32, + 0x2C60, 0x2C60, 32, 0x2C62, 0x2C64, 32, 0x2C67, 0x2C67, 32, 0x2C69, 0x2C69, 32, + 0x2C6B, 0x2C6B, 32, 0x2C6D, 0x2C70, 32, 0x2C72, 0x2C72, 32, 0x2C75, 0x2C75, 32, + 0x2C7E, 0x2C80, 32, 0x2C82, 0x2C82, 32, 0x2C84, 0x2C84, 32, 0x2C86, 0x2C86, 32, + 0x2C88, 0x2C88, 32, 0x2C8A, 0x2C8A, 32, 0x2C8C, 0x2C8C, 32, 0x2C8E, 0x2C8E, 32, + 0x2C90, 0x2C90, 32, 0x2C92, 0x2C92, 32, 0x2C94, 0x2C94, 32, 0x2C96, 0x2C96, 32, + 0x2C98, 0x2C98, 32, 0x2C9A, 0x2C9A, 32, 0x2C9C, 0x2C9C, 32, 0x2C9E, 0x2C9E, 32, + 0x2CA0, 0x2CA0, 32, 0x2CA2, 0x2CA2, 32, 0x2CA4, 0x2CA4, 32, 0x2CA6, 0x2CA6, 32, + 0x2CA8, 0x2CA8, 32, 0x2CAA, 0x2CAA, 32, 0x2CAC, 0x2CAC, 32, 0x2CAE, 0x2CAE, 32, + 0x2CB0, 0x2CB0, 32, 0x2CB2, 0x2CB2, 32, 0x2CB4, 0x2CB4, 32, 0x2CB6, 0x2CB6, 32, + 0x2CB8, 0x2CB8, 32, 0x2CBA, 0x2CBA, 32, 0x2CBC, 0x2CBC, 32, 0x2CBE, 0x2CBE, 32, + 0x2CC0, 0x2CC0, 32, 0x2CC2, 0x2CC2, 32, 0x2CC4, 0x2CC4, 32, 0x2CC6, 0x2CC6, 32, + 0x2CC8, 0x2CC8, 32, 0x2CCA, 0x2CCA, 32, 0x2CCC, 0x2CCC, 32, 0x2CCE, 0x2CCE, 32, + 0x2CD0, 0x2CD0, 32, 0x2CD2, 0x2CD2, 32, 0x2CD4, 0x2CD4, 32, 0x2CD6, 0x2CD6, 32, + 0x2CD8, 0x2CD8, 32, 0x2CDA, 0x2CDA, 32, 0x2CDC, 0x2CDC, 32, 0x2CDE, 0x2CDE, 32, + 0x2CE0, 0x2CE0, 32, 0x2CE2, 0x2CE2, 32, 0x2CEB, 0x2CEB, 32, 0x2CED, 0x2CED, 32, + 0x2CF2, 0x2CF2, 32, 0xA640, 0xA640, 32, 0xA642, 0xA642, 32, 0xA644, 0xA644, 32, + 0xA646, 0xA646, 32, 0xA648, 0xA648, 32, 0xA64A, 0xA64A, 32, 0xA64C, 0xA64C, 32, + 0xA64E, 0xA64E, 32, 0xA650, 0xA650, 32, 0xA652, 0xA652, 32, 0xA654, 0xA654, 32, + 0xA656, 0xA656, 32, 0xA658, 0xA658, 32, 0xA65A, 0xA65A, 32, 0xA65C, 0xA65C, 32, + 0xA65E, 0xA65E, 32, 0xA660, 0xA660, 32, 0xA662, 0xA662, 32, 0xA664, 0xA664, 32, + 0xA666, 0xA666, 32, 0xA668, 0xA668, 32, 0xA66A, 0xA66A, 32, 0xA66C, 0xA66C, 32, + 0xA680, 0xA680, 32, 0xA682, 0xA682, 32, 0xA684, 0xA684, 32, 0xA686, 0xA686, 32, + 0xA688, 0xA688, 32, 0xA68A, 0xA68A, 32, 0xA68C, 0xA68C, 32, 0xA68E, 0xA68E, 32, + 0xA690, 0xA690, 32, 0xA692, 0xA692, 32, 0xA694, 0xA694, 32, 0xA696, 0xA696, 32, + 0xA698, 0xA698, 32, 0xA69A, 0xA69A, 32, 0xA722, 0xA722, 32, 0xA724, 0xA724, 32, + 0xA726, 0xA726, 32, 0xA728, 0xA728, 32, 0xA72A, 0xA72A, 32, 0xA72C, 0xA72C, 32, + 0xA72E, 0xA72E, 32, 0xA732, 0xA732, 32, 0xA734, 0xA734, 32, 0xA736, 0xA736, 32, + 0xA738, 0xA738, 32, 0xA73A, 0xA73A, 32, 0xA73C, 0xA73C, 32, 0xA73E, 0xA73E, 32, + 0xA740, 0xA740, 32, 0xA742, 0xA742, 32, 0xA744, 0xA744, 32, 0xA746, 0xA746, 32, + 0xA748, 0xA748, 32, 0xA74A, 0xA74A, 32, 0xA74C, 0xA74C, 32, 0xA74E, 0xA74E, 32, + 0xA750, 0xA750, 32, 0xA752, 0xA752, 32, 0xA754, 0xA754, 32, 0xA756, 0xA756, 32, + 0xA758, 0xA758, 32, 0xA75A, 0xA75A, 32, 0xA75C, 0xA75C, 32, 0xA75E, 0xA75E, 32, + 0xA760, 0xA760, 32, 0xA762, 0xA762, 32, 0xA764, 0xA764, 32, 0xA766, 0xA766, 32, + 0xA768, 0xA768, 32, 0xA76A, 0xA76A, 32, 0xA76C, 0xA76C, 32, 0xA76E, 0xA76E, 32, + 0xA779, 0xA779, 32, 0xA77B, 0xA77B, 32, 0xA77D, 0xA77E, 32, 0xA780, 0xA780, 32, + 0xA782, 0xA782, 32, 0xA784, 0xA784, 32, 0xA786, 0xA786, 32, 0xA78B, 0xA78B, 32, + 0xA78D, 0xA78D, 32, 0xA790, 0xA790, 32, 0xA792, 0xA792, 32, 0xA796, 0xA796, 32, + 0xA798, 0xA798, 32, 0xA79A, 0xA79A, 32, 0xA79C, 0xA79C, 32, 0xA79E, 0xA79E, 32, + 0xA7A0, 0xA7A0, 32, 0xA7A2, 0xA7A2, 32, 0xA7A4, 0xA7A4, 32, 0xA7A6, 0xA7A6, 32, + 0xA7A8, 0xA7A8, 32, 0xA7AA, 0xA7AE, 32, 0xA7B0, 0xA7B4, 32, 0xA7B6, 0xA7B6, 32, + 0xA7B8, 0xA7B8, 32, 0xA7BA, 0xA7BA, 32, 0xA7BC, 0xA7BC, 32, 0xA7BE, 0xA7BE, 32, + 0xA7C0, 0xA7C0, 32, 0xA7C2, 0xA7C2, 32, 0xA7C4, 0xA7C7, 32, 0xA7C9, 0xA7C9, 32, + 0xA7CB, 0xA7CC, 32, 0xA7CE, 0xA7CE, 32, 0xA7D0, 0xA7D0, 32, 0xA7D2, 0xA7D2, 32, + 0xA7D4, 0xA7D4, 32, 0xA7D6, 0xA7D6, 32, 0xA7D8, 0xA7D8, 32, 0xA7DA, 0xA7DA, 32, + 0xA7DC, 0xA7DC, 32, 0xA7F5, 0xA7F5, 32, 0xFF21, 0xFF3A, 32, 0x10400, 0x10427, 32, + 0x104B0, 0x104D3, 32, 0x10570, 0x1057A, 32, 0x1057C, 0x1058A, 32, 0x1058C, 0x10592, 32, + 0x10594, 0x10595, 32, 0x10C80, 0x10CB2, 32, 0x10D50, 0x10D65, 32, 0x118A0, 0x118BF, 32, + 0x16E40, 0x16E5F, 32, 0x16EA0, 0x16EB8, 32, 0x1D400, 0x1D419, 32, 0x1D434, 0x1D44D, 32, + 0x1D468, 0x1D481, 32, 0x1D49C, 0x1D49C, 32, 0x1D49E, 0x1D49F, 32, 0x1D4A2, 0x1D4A2, 32, + 0x1D4A5, 0x1D4A6, 32, 0x1D4A9, 0x1D4AC, 32, 0x1D4AE, 0x1D4B5, 32, 0x1D4D0, 0x1D4E9, 32, + 0x1D504, 0x1D505, 32, 0x1D507, 0x1D50A, 32, 0x1D50D, 0x1D514, 32, 0x1D516, 0x1D51C, 32, + 0x1D538, 0x1D539, 32, 0x1D53B, 0x1D53E, 32, 0x1D540, 0x1D544, 32, 0x1D546, 0x1D546, 32, + 0x1D54A, 0x1D550, 32, 0x1D56C, 0x1D585, 32, 0x1D5A0, 0x1D5B9, 32, 0x1D5D4, 0x1D5ED, 32, + 0x1D608, 0x1D621, 32, 0x1D63C, 0x1D655, 32, 0x1D670, 0x1D689, 32, 0x1D6A8, 0x1D6C0, 32, + 0x1D6E2, 0x1D6FA, 32, 0x1D71C, 0x1D734, 32, 0x1D756, 0x1D76E, 32, 0x1D790, 0x1D7A8, 32, + 0x1D7CA, 0x1D7CA, 32, 0x1E900, 0x1E921, 32, 0x61, 0x7A, 32, 0xB5, 0xB5, 32, + 0xDF, 0xF6, 32, 0xF8, 0xFF, 32, 0x101, 0x101, 32, 0x103, 0x103, 32, + 0x105, 0x105, 32, 0x107, 0x107, 32, 0x109, 0x109, 32, 0x10B, 0x10B, 32, + 0x10D, 0x10D, 32, 0x10F, 0x10F, 32, 0x111, 0x111, 32, 0x113, 0x113, 32, + 0x115, 0x115, 32, 0x117, 0x117, 32, 0x119, 0x119, 32, 0x11B, 0x11B, 32, + 0x11D, 0x11D, 32, 0x11F, 0x11F, 32, 0x121, 0x121, 32, 0x123, 0x123, 32, + 0x125, 0x125, 32, 0x127, 0x127, 32, 0x129, 0x129, 32, 0x12B, 0x12B, 32, + 0x12D, 0x12D, 32, 0x12F, 0x12F, 32, 0x131, 0x131, 32, 0x133, 0x133, 32, + 0x135, 0x135, 32, 0x137, 0x138, 32, 0x13A, 0x13A, 32, 0x13C, 0x13C, 32, + 0x13E, 0x13E, 32, 0x140, 0x140, 32, 0x142, 0x142, 32, 0x144, 0x144, 32, + 0x146, 0x146, 32, 0x148, 0x149, 32, 0x14B, 0x14B, 32, 0x14D, 0x14D, 32, + 0x14F, 0x14F, 32, 0x151, 0x151, 32, 0x153, 0x153, 32, 0x155, 0x155, 32, + 0x157, 0x157, 32, 0x159, 0x159, 32, 0x15B, 0x15B, 32, 0x15D, 0x15D, 32, + 0x15F, 0x15F, 32, 0x161, 0x161, 32, 0x163, 0x163, 32, 0x165, 0x165, 32, + 0x167, 0x167, 32, 0x169, 0x169, 32, 0x16B, 0x16B, 32, 0x16D, 0x16D, 32, + 0x16F, 0x16F, 32, 0x171, 0x171, 32, 0x173, 0x173, 32, 0x175, 0x175, 32, + 0x177, 0x177, 32, 0x17A, 0x17A, 32, 0x17C, 0x17C, 32, 0x17E, 0x180, 32, + 0x183, 0x183, 32, 0x185, 0x185, 32, 0x188, 0x188, 32, 0x18C, 0x18D, 32, + 0x192, 0x192, 32, 0x195, 0x195, 32, 0x199, 0x19B, 32, 0x19E, 0x19E, 32, + 0x1A1, 0x1A1, 32, 0x1A3, 0x1A3, 32, 0x1A5, 0x1A5, 32, 0x1A8, 0x1A8, 32, + 0x1AA, 0x1AB, 32, 0x1AD, 0x1AD, 32, 0x1B0, 0x1B0, 32, 0x1B4, 0x1B4, 32, + 0x1B6, 0x1B6, 32, 0x1B9, 0x1BA, 32, 0x1BD, 0x1BF, 32, 0x1C6, 0x1C6, 32, + 0x1C9, 0x1C9, 32, 0x1CC, 0x1CC, 32, 0x1CE, 0x1CE, 32, 0x1D0, 0x1D0, 32, + 0x1D2, 0x1D2, 32, 0x1D4, 0x1D4, 32, 0x1D6, 0x1D6, 32, 0x1D8, 0x1D8, 32, + }; + } + + private static int[] rangeChunk19() { + return new int[] { + 0x1DA, 0x1DA, 32, 0x1DC, 0x1DD, 32, 0x1DF, 0x1DF, 32, 0x1E1, 0x1E1, 32, + 0x1E3, 0x1E3, 32, 0x1E5, 0x1E5, 32, 0x1E7, 0x1E7, 32, 0x1E9, 0x1E9, 32, + 0x1EB, 0x1EB, 32, 0x1ED, 0x1ED, 32, 0x1EF, 0x1F0, 32, 0x1F3, 0x1F3, 32, + 0x1F5, 0x1F5, 32, 0x1F9, 0x1F9, 32, 0x1FB, 0x1FB, 32, 0x1FD, 0x1FD, 32, + 0x1FF, 0x1FF, 32, 0x201, 0x201, 32, 0x203, 0x203, 32, 0x205, 0x205, 32, + 0x207, 0x207, 32, 0x209, 0x209, 32, 0x20B, 0x20B, 32, 0x20D, 0x20D, 32, + 0x20F, 0x20F, 32, 0x211, 0x211, 32, 0x213, 0x213, 32, 0x215, 0x215, 32, + 0x217, 0x217, 32, 0x219, 0x219, 32, 0x21B, 0x21B, 32, 0x21D, 0x21D, 32, + 0x21F, 0x21F, 32, 0x221, 0x221, 32, 0x223, 0x223, 32, 0x225, 0x225, 32, + 0x227, 0x227, 32, 0x229, 0x229, 32, 0x22B, 0x22B, 32, 0x22D, 0x22D, 32, + 0x22F, 0x22F, 32, 0x231, 0x231, 32, 0x233, 0x239, 32, 0x23C, 0x23C, 32, + 0x23F, 0x240, 32, 0x242, 0x242, 32, 0x247, 0x247, 32, 0x249, 0x249, 32, + 0x24B, 0x24B, 32, 0x24D, 0x24D, 32, 0x24F, 0x293, 32, 0x296, 0x2AF, 32, + 0x371, 0x371, 32, 0x373, 0x373, 32, 0x377, 0x377, 32, 0x37B, 0x37D, 32, + 0x390, 0x390, 32, 0x3AC, 0x3CE, 32, 0x3D0, 0x3D1, 32, 0x3D5, 0x3D7, 32, + 0x3D9, 0x3D9, 32, 0x3DB, 0x3DB, 32, 0x3DD, 0x3DD, 32, 0x3DF, 0x3DF, 32, + 0x3E1, 0x3E1, 32, 0x3E3, 0x3E3, 32, 0x3E5, 0x3E5, 32, 0x3E7, 0x3E7, 32, + 0x3E9, 0x3E9, 32, 0x3EB, 0x3EB, 32, 0x3ED, 0x3ED, 32, 0x3EF, 0x3F3, 32, + 0x3F5, 0x3F5, 32, 0x3F8, 0x3F8, 32, 0x3FB, 0x3FC, 32, 0x430, 0x45F, 32, + 0x461, 0x461, 32, 0x463, 0x463, 32, 0x465, 0x465, 32, 0x467, 0x467, 32, + 0x469, 0x469, 32, 0x46B, 0x46B, 32, 0x46D, 0x46D, 32, 0x46F, 0x46F, 32, + 0x471, 0x471, 32, 0x473, 0x473, 32, 0x475, 0x475, 32, 0x477, 0x477, 32, + 0x479, 0x479, 32, 0x47B, 0x47B, 32, 0x47D, 0x47D, 32, 0x47F, 0x47F, 32, + 0x481, 0x481, 32, 0x48B, 0x48B, 32, 0x48D, 0x48D, 32, 0x48F, 0x48F, 32, + 0x491, 0x491, 32, 0x493, 0x493, 32, 0x495, 0x495, 32, 0x497, 0x497, 32, + 0x499, 0x499, 32, 0x49B, 0x49B, 32, 0x49D, 0x49D, 32, 0x49F, 0x49F, 32, + 0x4A1, 0x4A1, 32, 0x4A3, 0x4A3, 32, 0x4A5, 0x4A5, 32, 0x4A7, 0x4A7, 32, + 0x4A9, 0x4A9, 32, 0x4AB, 0x4AB, 32, 0x4AD, 0x4AD, 32, 0x4AF, 0x4AF, 32, + 0x4B1, 0x4B1, 32, 0x4B3, 0x4B3, 32, 0x4B5, 0x4B5, 32, 0x4B7, 0x4B7, 32, + 0x4B9, 0x4B9, 32, 0x4BB, 0x4BB, 32, 0x4BD, 0x4BD, 32, 0x4BF, 0x4BF, 32, + 0x4C2, 0x4C2, 32, 0x4C4, 0x4C4, 32, 0x4C6, 0x4C6, 32, 0x4C8, 0x4C8, 32, + 0x4CA, 0x4CA, 32, 0x4CC, 0x4CC, 32, 0x4CE, 0x4CF, 32, 0x4D1, 0x4D1, 32, + 0x4D3, 0x4D3, 32, 0x4D5, 0x4D5, 32, 0x4D7, 0x4D7, 32, 0x4D9, 0x4D9, 32, + 0x4DB, 0x4DB, 32, 0x4DD, 0x4DD, 32, 0x4DF, 0x4DF, 32, 0x4E1, 0x4E1, 32, + 0x4E3, 0x4E3, 32, 0x4E5, 0x4E5, 32, 0x4E7, 0x4E7, 32, 0x4E9, 0x4E9, 32, + 0x4EB, 0x4EB, 32, 0x4ED, 0x4ED, 32, 0x4EF, 0x4EF, 32, 0x4F1, 0x4F1, 32, + 0x4F3, 0x4F3, 32, 0x4F5, 0x4F5, 32, 0x4F7, 0x4F7, 32, 0x4F9, 0x4F9, 32, + 0x4FB, 0x4FB, 32, 0x4FD, 0x4FD, 32, 0x4FF, 0x4FF, 32, 0x501, 0x501, 32, + 0x503, 0x503, 32, 0x505, 0x505, 32, 0x507, 0x507, 32, 0x509, 0x509, 32, + 0x50B, 0x50B, 32, 0x50D, 0x50D, 32, 0x50F, 0x50F, 32, 0x511, 0x511, 32, + 0x513, 0x513, 32, 0x515, 0x515, 32, 0x517, 0x517, 32, 0x519, 0x519, 32, + 0x51B, 0x51B, 32, 0x51D, 0x51D, 32, 0x51F, 0x51F, 32, 0x521, 0x521, 32, + 0x523, 0x523, 32, 0x525, 0x525, 32, 0x527, 0x527, 32, 0x529, 0x529, 32, + 0x52B, 0x52B, 32, 0x52D, 0x52D, 32, 0x52F, 0x52F, 32, 0x560, 0x588, 32, + 0x10D0, 0x10FA, 32, 0x10FD, 0x10FF, 32, 0x13F8, 0x13FD, 32, 0x1C80, 0x1C88, 32, + 0x1C8A, 0x1C8A, 32, 0x1D00, 0x1D2B, 32, 0x1D6B, 0x1D77, 32, 0x1D79, 0x1D9A, 32, + 0x1E01, 0x1E01, 32, 0x1E03, 0x1E03, 32, 0x1E05, 0x1E05, 32, 0x1E07, 0x1E07, 32, + 0x1E09, 0x1E09, 32, 0x1E0B, 0x1E0B, 32, 0x1E0D, 0x1E0D, 32, 0x1E0F, 0x1E0F, 32, + 0x1E11, 0x1E11, 32, 0x1E13, 0x1E13, 32, 0x1E15, 0x1E15, 32, 0x1E17, 0x1E17, 32, + 0x1E19, 0x1E19, 32, 0x1E1B, 0x1E1B, 32, 0x1E1D, 0x1E1D, 32, 0x1E1F, 0x1E1F, 32, + 0x1E21, 0x1E21, 32, 0x1E23, 0x1E23, 32, 0x1E25, 0x1E25, 32, 0x1E27, 0x1E27, 32, + 0x1E29, 0x1E29, 32, 0x1E2B, 0x1E2B, 32, 0x1E2D, 0x1E2D, 32, 0x1E2F, 0x1E2F, 32, + 0x1E31, 0x1E31, 32, 0x1E33, 0x1E33, 32, 0x1E35, 0x1E35, 32, 0x1E37, 0x1E37, 32, + 0x1E39, 0x1E39, 32, 0x1E3B, 0x1E3B, 32, 0x1E3D, 0x1E3D, 32, 0x1E3F, 0x1E3F, 32, + 0x1E41, 0x1E41, 32, 0x1E43, 0x1E43, 32, 0x1E45, 0x1E45, 32, 0x1E47, 0x1E47, 32, + 0x1E49, 0x1E49, 32, 0x1E4B, 0x1E4B, 32, 0x1E4D, 0x1E4D, 32, 0x1E4F, 0x1E4F, 32, + 0x1E51, 0x1E51, 32, 0x1E53, 0x1E53, 32, 0x1E55, 0x1E55, 32, 0x1E57, 0x1E57, 32, + 0x1E59, 0x1E59, 32, 0x1E5B, 0x1E5B, 32, 0x1E5D, 0x1E5D, 32, 0x1E5F, 0x1E5F, 32, + 0x1E61, 0x1E61, 32, 0x1E63, 0x1E63, 32, 0x1E65, 0x1E65, 32, 0x1E67, 0x1E67, 32, + 0x1E69, 0x1E69, 32, 0x1E6B, 0x1E6B, 32, 0x1E6D, 0x1E6D, 32, 0x1E6F, 0x1E6F, 32, + 0x1E71, 0x1E71, 32, 0x1E73, 0x1E73, 32, 0x1E75, 0x1E75, 32, 0x1E77, 0x1E77, 32, + 0x1E79, 0x1E79, 32, 0x1E7B, 0x1E7B, 32, 0x1E7D, 0x1E7D, 32, 0x1E7F, 0x1E7F, 32, + 0x1E81, 0x1E81, 32, 0x1E83, 0x1E83, 32, 0x1E85, 0x1E85, 32, 0x1E87, 0x1E87, 32, + 0x1E89, 0x1E89, 32, 0x1E8B, 0x1E8B, 32, 0x1E8D, 0x1E8D, 32, 0x1E8F, 0x1E8F, 32, + 0x1E91, 0x1E91, 32, 0x1E93, 0x1E93, 32, 0x1E95, 0x1E9D, 32, 0x1E9F, 0x1E9F, 32, + 0x1EA1, 0x1EA1, 32, 0x1EA3, 0x1EA3, 32, 0x1EA5, 0x1EA5, 32, 0x1EA7, 0x1EA7, 32, + 0x1EA9, 0x1EA9, 32, 0x1EAB, 0x1EAB, 32, 0x1EAD, 0x1EAD, 32, 0x1EAF, 0x1EAF, 32, + 0x1EB1, 0x1EB1, 32, 0x1EB3, 0x1EB3, 32, 0x1EB5, 0x1EB5, 32, 0x1EB7, 0x1EB7, 32, + 0x1EB9, 0x1EB9, 32, 0x1EBB, 0x1EBB, 32, 0x1EBD, 0x1EBD, 32, 0x1EBF, 0x1EBF, 32, + 0x1EC1, 0x1EC1, 32, 0x1EC3, 0x1EC3, 32, 0x1EC5, 0x1EC5, 32, 0x1EC7, 0x1EC7, 32, + 0x1EC9, 0x1EC9, 32, 0x1ECB, 0x1ECB, 32, 0x1ECD, 0x1ECD, 32, 0x1ECF, 0x1ECF, 32, + 0x1ED1, 0x1ED1, 32, 0x1ED3, 0x1ED3, 32, 0x1ED5, 0x1ED5, 32, 0x1ED7, 0x1ED7, 32, + 0x1ED9, 0x1ED9, 32, 0x1EDB, 0x1EDB, 32, 0x1EDD, 0x1EDD, 32, 0x1EDF, 0x1EDF, 32, + 0x1EE1, 0x1EE1, 32, 0x1EE3, 0x1EE3, 32, 0x1EE5, 0x1EE5, 32, 0x1EE7, 0x1EE7, 32, + 0x1EE9, 0x1EE9, 32, 0x1EEB, 0x1EEB, 32, 0x1EED, 0x1EED, 32, 0x1EEF, 0x1EEF, 32, + 0x1EF1, 0x1EF1, 32, 0x1EF3, 0x1EF3, 32, 0x1EF5, 0x1EF5, 32, 0x1EF7, 0x1EF7, 32, + 0x1EF9, 0x1EF9, 32, 0x1EFB, 0x1EFB, 32, 0x1EFD, 0x1EFD, 32, 0x1EFF, 0x1F07, 32, + 0x1F10, 0x1F15, 32, 0x1F20, 0x1F27, 32, 0x1F30, 0x1F37, 32, 0x1F40, 0x1F45, 32, + 0x1F50, 0x1F57, 32, 0x1F60, 0x1F67, 32, 0x1F70, 0x1F7D, 32, 0x1F80, 0x1F87, 32, + 0x1F90, 0x1F97, 32, 0x1FA0, 0x1FA7, 32, 0x1FB0, 0x1FB4, 32, 0x1FB6, 0x1FB7, 32, + 0x1FBE, 0x1FBE, 32, 0x1FC2, 0x1FC4, 32, 0x1FC6, 0x1FC7, 32, 0x1FD0, 0x1FD3, 32, + 0x1FD6, 0x1FD7, 32, 0x1FE0, 0x1FE7, 32, 0x1FF2, 0x1FF4, 32, 0x1FF6, 0x1FF7, 32, + 0x210A, 0x210A, 32, 0x210E, 0x210F, 32, 0x2113, 0x2113, 32, 0x212F, 0x212F, 32, + 0x2134, 0x2134, 32, 0x2139, 0x2139, 32, 0x213C, 0x213D, 32, 0x2146, 0x2149, 32, + 0x214E, 0x214E, 32, 0x2184, 0x2184, 32, 0x2C30, 0x2C5F, 32, 0x2C61, 0x2C61, 32, + 0x2C65, 0x2C66, 32, 0x2C68, 0x2C68, 32, 0x2C6A, 0x2C6A, 32, 0x2C6C, 0x2C6C, 32, + 0x2C71, 0x2C71, 32, 0x2C73, 0x2C74, 32, 0x2C76, 0x2C7B, 32, 0x2C81, 0x2C81, 32, + 0x2C83, 0x2C83, 32, 0x2C85, 0x2C85, 32, 0x2C87, 0x2C87, 32, 0x2C89, 0x2C89, 32, + 0x2C8B, 0x2C8B, 32, 0x2C8D, 0x2C8D, 32, 0x2C8F, 0x2C8F, 32, 0x2C91, 0x2C91, 32, + 0x2C93, 0x2C93, 32, 0x2C95, 0x2C95, 32, 0x2C97, 0x2C97, 32, 0x2C99, 0x2C99, 32, + 0x2C9B, 0x2C9B, 32, 0x2C9D, 0x2C9D, 32, 0x2C9F, 0x2C9F, 32, 0x2CA1, 0x2CA1, 32, + 0x2CA3, 0x2CA3, 32, 0x2CA5, 0x2CA5, 32, 0x2CA7, 0x2CA7, 32, 0x2CA9, 0x2CA9, 32, + 0x2CAB, 0x2CAB, 32, 0x2CAD, 0x2CAD, 32, 0x2CAF, 0x2CAF, 32, 0x2CB1, 0x2CB1, 32, + 0x2CB3, 0x2CB3, 32, 0x2CB5, 0x2CB5, 32, 0x2CB7, 0x2CB7, 32, 0x2CB9, 0x2CB9, 32, + 0x2CBB, 0x2CBB, 32, 0x2CBD, 0x2CBD, 32, 0x2CBF, 0x2CBF, 32, 0x2CC1, 0x2CC1, 32, + 0x2CC3, 0x2CC3, 32, 0x2CC5, 0x2CC5, 32, 0x2CC7, 0x2CC7, 32, 0x2CC9, 0x2CC9, 32, + 0x2CCB, 0x2CCB, 32, 0x2CCD, 0x2CCD, 32, 0x2CCF, 0x2CCF, 32, 0x2CD1, 0x2CD1, 32, + 0x2CD3, 0x2CD3, 32, 0x2CD5, 0x2CD5, 32, 0x2CD7, 0x2CD7, 32, 0x2CD9, 0x2CD9, 32, + 0x2CDB, 0x2CDB, 32, 0x2CDD, 0x2CDD, 32, 0x2CDF, 0x2CDF, 32, 0x2CE1, 0x2CE1, 32, + 0x2CE3, 0x2CE4, 32, 0x2CEC, 0x2CEC, 32, 0x2CEE, 0x2CEE, 32, 0x2CF3, 0x2CF3, 32, + }; + } + + private static int[] rangeChunk20() { + return new int[] { + 0x2D00, 0x2D25, 32, 0x2D27, 0x2D27, 32, 0x2D2D, 0x2D2D, 32, 0xA641, 0xA641, 32, + 0xA643, 0xA643, 32, 0xA645, 0xA645, 32, 0xA647, 0xA647, 32, 0xA649, 0xA649, 32, + 0xA64B, 0xA64B, 32, 0xA64D, 0xA64D, 32, 0xA64F, 0xA64F, 32, 0xA651, 0xA651, 32, + 0xA653, 0xA653, 32, 0xA655, 0xA655, 32, 0xA657, 0xA657, 32, 0xA659, 0xA659, 32, + 0xA65B, 0xA65B, 32, 0xA65D, 0xA65D, 32, 0xA65F, 0xA65F, 32, 0xA661, 0xA661, 32, + 0xA663, 0xA663, 32, 0xA665, 0xA665, 32, 0xA667, 0xA667, 32, 0xA669, 0xA669, 32, + 0xA66B, 0xA66B, 32, 0xA66D, 0xA66D, 32, 0xA681, 0xA681, 32, 0xA683, 0xA683, 32, + 0xA685, 0xA685, 32, 0xA687, 0xA687, 32, 0xA689, 0xA689, 32, 0xA68B, 0xA68B, 32, + 0xA68D, 0xA68D, 32, 0xA68F, 0xA68F, 32, 0xA691, 0xA691, 32, 0xA693, 0xA693, 32, + 0xA695, 0xA695, 32, 0xA697, 0xA697, 32, 0xA699, 0xA699, 32, 0xA69B, 0xA69B, 32, + 0xA723, 0xA723, 32, 0xA725, 0xA725, 32, 0xA727, 0xA727, 32, 0xA729, 0xA729, 32, + 0xA72B, 0xA72B, 32, 0xA72D, 0xA72D, 32, 0xA72F, 0xA731, 32, 0xA733, 0xA733, 32, + 0xA735, 0xA735, 32, 0xA737, 0xA737, 32, 0xA739, 0xA739, 32, 0xA73B, 0xA73B, 32, + 0xA73D, 0xA73D, 32, 0xA73F, 0xA73F, 32, 0xA741, 0xA741, 32, 0xA743, 0xA743, 32, + 0xA745, 0xA745, 32, 0xA747, 0xA747, 32, 0xA749, 0xA749, 32, 0xA74B, 0xA74B, 32, + 0xA74D, 0xA74D, 32, 0xA74F, 0xA74F, 32, 0xA751, 0xA751, 32, 0xA753, 0xA753, 32, + 0xA755, 0xA755, 32, 0xA757, 0xA757, 32, 0xA759, 0xA759, 32, 0xA75B, 0xA75B, 32, + 0xA75D, 0xA75D, 32, 0xA75F, 0xA75F, 32, 0xA761, 0xA761, 32, 0xA763, 0xA763, 32, + 0xA765, 0xA765, 32, 0xA767, 0xA767, 32, 0xA769, 0xA769, 32, 0xA76B, 0xA76B, 32, + 0xA76D, 0xA76D, 32, 0xA76F, 0xA76F, 32, 0xA771, 0xA778, 32, 0xA77A, 0xA77A, 32, + 0xA77C, 0xA77C, 32, 0xA77F, 0xA77F, 32, 0xA781, 0xA781, 32, 0xA783, 0xA783, 32, + 0xA785, 0xA785, 32, 0xA787, 0xA787, 32, 0xA78C, 0xA78C, 32, 0xA78E, 0xA78E, 32, + 0xA791, 0xA791, 32, 0xA793, 0xA795, 32, 0xA797, 0xA797, 32, 0xA799, 0xA799, 32, + 0xA79B, 0xA79B, 32, 0xA79D, 0xA79D, 32, 0xA79F, 0xA79F, 32, 0xA7A1, 0xA7A1, 32, + 0xA7A3, 0xA7A3, 32, 0xA7A5, 0xA7A5, 32, 0xA7A7, 0xA7A7, 32, 0xA7A9, 0xA7A9, 32, + 0xA7AF, 0xA7AF, 32, 0xA7B5, 0xA7B5, 32, 0xA7B7, 0xA7B7, 32, 0xA7B9, 0xA7B9, 32, + 0xA7BB, 0xA7BB, 32, 0xA7BD, 0xA7BD, 32, 0xA7BF, 0xA7BF, 32, 0xA7C1, 0xA7C1, 32, + 0xA7C3, 0xA7C3, 32, 0xA7C8, 0xA7C8, 32, 0xA7CA, 0xA7CA, 32, 0xA7CD, 0xA7CD, 32, + 0xA7CF, 0xA7CF, 32, 0xA7D1, 0xA7D1, 32, 0xA7D3, 0xA7D3, 32, 0xA7D5, 0xA7D5, 32, + 0xA7D7, 0xA7D7, 32, 0xA7D9, 0xA7D9, 32, 0xA7DB, 0xA7DB, 32, 0xA7F6, 0xA7F6, 32, + 0xA7FA, 0xA7FA, 32, 0xAB30, 0xAB5A, 32, 0xAB60, 0xAB68, 32, 0xAB70, 0xABBF, 32, + 0xFB00, 0xFB06, 32, 0xFB13, 0xFB17, 32, 0xFF41, 0xFF5A, 32, 0x10428, 0x1044F, 32, + 0x104D8, 0x104FB, 32, 0x10597, 0x105A1, 32, 0x105A3, 0x105B1, 32, 0x105B3, 0x105B9, 32, + 0x105BB, 0x105BC, 32, 0x10CC0, 0x10CF2, 32, 0x10D70, 0x10D85, 32, 0x118C0, 0x118DF, 32, + 0x16E60, 0x16E7F, 32, 0x16EBB, 0x16ED3, 32, 0x1D41A, 0x1D433, 32, 0x1D44E, 0x1D454, 32, + 0x1D456, 0x1D467, 32, 0x1D482, 0x1D49B, 32, 0x1D4B6, 0x1D4B9, 32, 0x1D4BB, 0x1D4BB, 32, + 0x1D4BD, 0x1D4C3, 32, 0x1D4C5, 0x1D4CF, 32, 0x1D4EA, 0x1D503, 32, 0x1D51E, 0x1D537, 32, + 0x1D552, 0x1D56B, 32, 0x1D586, 0x1D59F, 32, 0x1D5BA, 0x1D5D3, 32, 0x1D5EE, 0x1D607, 32, + 0x1D622, 0x1D63B, 32, 0x1D656, 0x1D66F, 32, 0x1D68A, 0x1D6A5, 32, 0x1D6C2, 0x1D6DA, 32, + 0x1D6DC, 0x1D6E1, 32, 0x1D6FC, 0x1D714, 32, 0x1D716, 0x1D71B, 32, 0x1D736, 0x1D74E, 32, + 0x1D750, 0x1D755, 32, 0x1D770, 0x1D788, 32, 0x1D78A, 0x1D78F, 32, 0x1D7AA, 0x1D7C2, 32, + 0x1D7C4, 0x1D7C9, 32, 0x1D7CB, 0x1D7CB, 32, 0x1DF00, 0x1DF09, 32, 0x1DF0B, 0x1DF1E, 32, + 0x1DF25, 0x1DF2A, 32, 0x1E922, 0x1E943, 32, 0x1C5, 0x1C5, 32, 0x1C8, 0x1C8, 32, + 0x1CB, 0x1CB, 32, 0x1F2, 0x1F2, 32, 0x1F88, 0x1F8F, 32, 0x1F98, 0x1F9F, 32, + 0x1FA8, 0x1FAF, 32, 0x1FBC, 0x1FBC, 32, 0x1FCC, 0x1FCC, 32, 0x1FFC, 0x1FFC, 32, + 0x300, 0x36F, 33, 0x483, 0x487, 33, 0x591, 0x5BD, 33, 0x5BF, 0x5BF, 33, + 0x5C1, 0x5C2, 33, 0x5C4, 0x5C5, 33, 0x5C7, 0x5C7, 33, 0x610, 0x61A, 33, + 0x64B, 0x65F, 33, 0x670, 0x670, 33, 0x6D6, 0x6DC, 33, 0x6DF, 0x6E4, 33, + 0x6E7, 0x6E8, 33, 0x6EA, 0x6ED, 33, 0x711, 0x711, 33, 0x730, 0x74A, 33, + 0x7A6, 0x7B0, 33, 0x7EB, 0x7F3, 33, 0x7FD, 0x7FD, 33, 0x816, 0x819, 33, + 0x81B, 0x823, 33, 0x825, 0x827, 33, 0x829, 0x82D, 33, 0x859, 0x85B, 33, + 0x897, 0x89F, 33, 0x8CA, 0x8E1, 33, 0x8E3, 0x902, 33, 0x93A, 0x93A, 33, + 0x93C, 0x93C, 33, 0x941, 0x948, 33, 0x94D, 0x94D, 33, 0x951, 0x957, 33, + 0x962, 0x963, 33, 0x981, 0x981, 33, 0x9BC, 0x9BC, 33, 0x9C1, 0x9C4, 33, + 0x9CD, 0x9CD, 33, 0x9E2, 0x9E3, 33, 0x9FE, 0x9FE, 33, 0xA01, 0xA02, 33, + 0xA3C, 0xA3C, 33, 0xA41, 0xA42, 33, 0xA47, 0xA48, 33, 0xA4B, 0xA4D, 33, + 0xA51, 0xA51, 33, 0xA70, 0xA71, 33, 0xA75, 0xA75, 33, 0xA81, 0xA82, 33, + 0xABC, 0xABC, 33, 0xAC1, 0xAC5, 33, 0xAC7, 0xAC8, 33, 0xACD, 0xACD, 33, + 0xAE2, 0xAE3, 33, 0xAFA, 0xAFF, 33, 0xB01, 0xB01, 33, 0xB3C, 0xB3C, 33, + 0xB3F, 0xB3F, 33, 0xB41, 0xB44, 33, 0xB4D, 0xB4D, 33, 0xB55, 0xB56, 33, + 0xB62, 0xB63, 33, 0xB82, 0xB82, 33, 0xBC0, 0xBC0, 33, 0xBCD, 0xBCD, 33, + 0xC00, 0xC00, 33, 0xC04, 0xC04, 33, 0xC3C, 0xC3C, 33, 0xC3E, 0xC40, 33, + 0xC46, 0xC48, 33, 0xC4A, 0xC4D, 33, 0xC55, 0xC56, 33, 0xC62, 0xC63, 33, + 0xC81, 0xC81, 33, 0xCBC, 0xCBC, 33, 0xCBF, 0xCBF, 33, 0xCC6, 0xCC6, 33, + 0xCCC, 0xCCD, 33, 0xCE2, 0xCE3, 33, 0xD00, 0xD01, 33, 0xD3B, 0xD3C, 33, + 0xD41, 0xD44, 33, 0xD4D, 0xD4D, 33, 0xD62, 0xD63, 33, 0xD81, 0xD81, 33, + 0xDCA, 0xDCA, 33, 0xDD2, 0xDD4, 33, 0xDD6, 0xDD6, 33, 0xE31, 0xE31, 33, + 0xE34, 0xE3A, 33, 0xE47, 0xE4E, 33, 0xEB1, 0xEB1, 33, 0xEB4, 0xEBC, 33, + 0xEC8, 0xECE, 33, 0xF18, 0xF19, 33, 0xF35, 0xF35, 33, 0xF37, 0xF37, 33, + 0xF39, 0xF39, 33, 0xF71, 0xF7E, 33, 0xF80, 0xF84, 33, 0xF86, 0xF87, 33, + 0xF8D, 0xF97, 33, 0xF99, 0xFBC, 33, 0xFC6, 0xFC6, 33, 0x102D, 0x1030, 33, + 0x1032, 0x1037, 33, 0x1039, 0x103A, 33, 0x103D, 0x103E, 33, 0x1058, 0x1059, 33, + 0x105E, 0x1060, 33, 0x1071, 0x1074, 33, 0x1082, 0x1082, 33, 0x1085, 0x1086, 33, + 0x108D, 0x108D, 33, 0x109D, 0x109D, 33, 0x135D, 0x135F, 33, 0x1712, 0x1714, 33, + 0x1732, 0x1733, 33, 0x1752, 0x1753, 33, 0x1772, 0x1773, 33, 0x17B4, 0x17B5, 33, + 0x17B7, 0x17BD, 33, 0x17C6, 0x17C6, 33, 0x17C9, 0x17D3, 33, 0x17DD, 0x17DD, 33, + 0x180B, 0x180D, 33, 0x180F, 0x180F, 33, 0x1885, 0x1886, 33, 0x18A9, 0x18A9, 33, + 0x1920, 0x1922, 33, 0x1927, 0x1928, 33, 0x1932, 0x1932, 33, 0x1939, 0x193B, 33, + 0x1A17, 0x1A18, 33, 0x1A1B, 0x1A1B, 33, 0x1A56, 0x1A56, 33, 0x1A58, 0x1A5E, 33, + 0x1A60, 0x1A60, 33, 0x1A62, 0x1A62, 33, 0x1A65, 0x1A6C, 33, 0x1A73, 0x1A7C, 33, + 0x1A7F, 0x1A7F, 33, 0x1AB0, 0x1ABD, 33, 0x1ABF, 0x1ADD, 33, 0x1AE0, 0x1AEB, 33, + 0x1B00, 0x1B03, 33, 0x1B34, 0x1B34, 33, 0x1B36, 0x1B3A, 33, 0x1B3C, 0x1B3C, 33, + 0x1B42, 0x1B42, 33, 0x1B6B, 0x1B73, 33, 0x1B80, 0x1B81, 33, 0x1BA2, 0x1BA5, 33, + 0x1BA8, 0x1BA9, 33, 0x1BAB, 0x1BAD, 33, 0x1BE6, 0x1BE6, 33, 0x1BE8, 0x1BE9, 33, + 0x1BED, 0x1BED, 33, 0x1BEF, 0x1BF1, 33, 0x1C2C, 0x1C33, 33, 0x1C36, 0x1C37, 33, + 0x1CD0, 0x1CD2, 33, 0x1CD4, 0x1CE0, 33, 0x1CE2, 0x1CE8, 33, 0x1CED, 0x1CED, 33, + 0x1CF4, 0x1CF4, 33, 0x1CF8, 0x1CF9, 33, 0x1DC0, 0x1DFF, 33, 0x20D0, 0x20DC, 33, + 0x20E1, 0x20E1, 33, 0x20E5, 0x20F0, 33, 0x2CEF, 0x2CF1, 33, 0x2D7F, 0x2D7F, 33, + 0x2DE0, 0x2DFF, 33, 0x302A, 0x302D, 33, 0x3099, 0x309A, 33, 0xA66F, 0xA66F, 33, + 0xA674, 0xA67D, 33, 0xA69E, 0xA69F, 33, 0xA6F0, 0xA6F1, 33, 0xA802, 0xA802, 33, + 0xA806, 0xA806, 33, 0xA80B, 0xA80B, 33, 0xA825, 0xA826, 33, 0xA82C, 0xA82C, 33, + 0xA8C4, 0xA8C5, 33, 0xA8E0, 0xA8F1, 33, 0xA8FF, 0xA8FF, 33, 0xA926, 0xA92D, 33, + 0xA947, 0xA951, 33, 0xA980, 0xA982, 33, 0xA9B3, 0xA9B3, 33, 0xA9B6, 0xA9B9, 33, + 0xA9BC, 0xA9BD, 33, 0xA9E5, 0xA9E5, 33, 0xAA29, 0xAA2E, 33, 0xAA31, 0xAA32, 33, + 0xAA35, 0xAA36, 33, 0xAA43, 0xAA43, 33, 0xAA4C, 0xAA4C, 33, 0xAA7C, 0xAA7C, 33, + 0xAAB0, 0xAAB0, 33, 0xAAB2, 0xAAB4, 33, 0xAAB7, 0xAAB8, 33, 0xAABE, 0xAABF, 33, + 0xAAC1, 0xAAC1, 33, 0xAAEC, 0xAAED, 33, 0xAAF6, 0xAAF6, 33, 0xABE5, 0xABE5, 33, + 0xABE8, 0xABE8, 33, 0xABED, 0xABED, 33, 0xFB1E, 0xFB1E, 33, 0xFE00, 0xFE0F, 33, + 0xFE20, 0xFE2F, 33, 0x101FD, 0x101FD, 33, 0x102E0, 0x102E0, 33, 0x10376, 0x1037A, 33, + 0x10A01, 0x10A03, 33, 0x10A05, 0x10A06, 33, 0x10A0C, 0x10A0F, 33, 0x10A38, 0x10A3A, 33, + }; + } + + private static int[] rangeChunk21() { + return new int[] { + 0x10A3F, 0x10A3F, 33, 0x10AE5, 0x10AE6, 33, 0x10D24, 0x10D27, 33, 0x10D69, 0x10D6D, 33, + 0x10EAB, 0x10EAC, 33, 0x10EFA, 0x10EFF, 33, 0x10F46, 0x10F50, 33, 0x10F82, 0x10F85, 33, + 0x11001, 0x11001, 33, 0x11038, 0x11046, 33, 0x11070, 0x11070, 33, 0x11073, 0x11074, 33, + 0x1107F, 0x11081, 33, 0x110B3, 0x110B6, 33, 0x110B9, 0x110BA, 33, 0x110C2, 0x110C2, 33, + 0x11100, 0x11102, 33, 0x11127, 0x1112B, 33, 0x1112D, 0x11134, 33, 0x11173, 0x11173, 33, + 0x11180, 0x11181, 33, 0x111B6, 0x111BE, 33, 0x111C9, 0x111CC, 33, 0x111CF, 0x111CF, 33, + 0x1122F, 0x11231, 33, 0x11234, 0x11234, 33, 0x11236, 0x11237, 33, 0x1123E, 0x1123E, 33, + 0x11241, 0x11241, 33, 0x112DF, 0x112DF, 33, 0x112E3, 0x112EA, 33, 0x11300, 0x11301, 33, + 0x1133B, 0x1133C, 33, 0x11340, 0x11340, 33, 0x11366, 0x1136C, 33, 0x11370, 0x11374, 33, + 0x113BB, 0x113C0, 33, 0x113CE, 0x113CE, 33, 0x113D0, 0x113D0, 33, 0x113D2, 0x113D2, 33, + 0x113E1, 0x113E2, 33, 0x11438, 0x1143F, 33, 0x11442, 0x11444, 33, 0x11446, 0x11446, 33, + 0x1145E, 0x1145E, 33, 0x114B3, 0x114B8, 33, 0x114BA, 0x114BA, 33, 0x114BF, 0x114C0, 33, + 0x114C2, 0x114C3, 33, 0x115B2, 0x115B5, 33, 0x115BC, 0x115BD, 33, 0x115BF, 0x115C0, 33, + 0x115DC, 0x115DD, 33, 0x11633, 0x1163A, 33, 0x1163D, 0x1163D, 33, 0x1163F, 0x11640, 33, + 0x116AB, 0x116AB, 33, 0x116AD, 0x116AD, 33, 0x116B0, 0x116B5, 33, 0x116B7, 0x116B7, 33, + 0x1171D, 0x1171D, 33, 0x1171F, 0x1171F, 33, 0x11722, 0x11725, 33, 0x11727, 0x1172B, 33, + 0x1182F, 0x11837, 33, 0x11839, 0x1183A, 33, 0x1193B, 0x1193C, 33, 0x1193E, 0x1193E, 33, + 0x11943, 0x11943, 33, 0x119D4, 0x119D7, 33, 0x119DA, 0x119DB, 33, 0x119E0, 0x119E0, 33, + 0x11A01, 0x11A0A, 33, 0x11A33, 0x11A38, 33, 0x11A3B, 0x11A3E, 33, 0x11A47, 0x11A47, 33, + 0x11A51, 0x11A56, 33, 0x11A59, 0x11A5B, 33, 0x11A8A, 0x11A96, 33, 0x11A98, 0x11A99, 33, + 0x11B60, 0x11B60, 33, 0x11B62, 0x11B64, 33, 0x11B66, 0x11B66, 33, 0x11C30, 0x11C36, 33, + 0x11C38, 0x11C3D, 33, 0x11C3F, 0x11C3F, 33, 0x11C92, 0x11CA7, 33, 0x11CAA, 0x11CB0, 33, + 0x11CB2, 0x11CB3, 33, 0x11CB5, 0x11CB6, 33, 0x11D31, 0x11D36, 33, 0x11D3A, 0x11D3A, 33, + 0x11D3C, 0x11D3D, 33, 0x11D3F, 0x11D45, 33, 0x11D47, 0x11D47, 33, 0x11D90, 0x11D91, 33, + 0x11D95, 0x11D95, 33, 0x11D97, 0x11D97, 33, 0x11EF3, 0x11EF4, 33, 0x11F00, 0x11F01, 33, + 0x11F36, 0x11F3A, 33, 0x11F40, 0x11F40, 33, 0x11F42, 0x11F42, 33, 0x11F5A, 0x11F5A, 33, + 0x13440, 0x13440, 33, 0x13447, 0x13455, 33, 0x1611E, 0x16129, 33, 0x1612D, 0x1612F, 33, + 0x16AF0, 0x16AF4, 33, 0x16B30, 0x16B36, 33, 0x16F4F, 0x16F4F, 33, 0x16F8F, 0x16F92, 33, + 0x16FE4, 0x16FE4, 33, 0x1BC9D, 0x1BC9E, 33, 0x1CF00, 0x1CF2D, 33, 0x1CF30, 0x1CF46, 33, + 0x1D167, 0x1D169, 33, 0x1D17B, 0x1D182, 33, 0x1D185, 0x1D18B, 33, 0x1D1AA, 0x1D1AD, 33, + 0x1D242, 0x1D244, 33, 0x1DA00, 0x1DA36, 33, 0x1DA3B, 0x1DA6C, 33, 0x1DA75, 0x1DA75, 33, + 0x1DA84, 0x1DA84, 33, 0x1DA9B, 0x1DA9F, 33, 0x1DAA1, 0x1DAAF, 33, 0x1E000, 0x1E006, 33, + 0x1E008, 0x1E018, 33, 0x1E01B, 0x1E021, 33, 0x1E023, 0x1E024, 33, 0x1E026, 0x1E02A, 33, + 0x1E08F, 0x1E08F, 33, 0x1E130, 0x1E136, 33, 0x1E2AE, 0x1E2AE, 33, 0x1E2EC, 0x1E2EF, 33, + 0x1E4EC, 0x1E4EF, 33, 0x1E5EE, 0x1E5EF, 33, 0x1E6E3, 0x1E6E3, 33, 0x1E6E6, 0x1E6E6, 33, + 0x1E6EE, 0x1E6EF, 33, 0x1E6F5, 0x1E6F5, 33, 0x1E8D0, 0x1E8D6, 33, 0x1E944, 0x1E94A, 33, + 0xE0100, 0xE01EF, 33, 0x488, 0x489, 33, 0x1ABE, 0x1ABE, 33, 0x20DD, 0x20E0, 33, + 0x20E2, 0x20E4, 33, 0xA670, 0xA672, 33, 0x903, 0x903, 33, 0x93B, 0x93B, 33, + 0x93E, 0x940, 33, 0x949, 0x94C, 33, 0x94E, 0x94F, 33, 0x982, 0x983, 33, + 0x9BE, 0x9C0, 33, 0x9C7, 0x9C8, 33, 0x9CB, 0x9CC, 33, 0x9D7, 0x9D7, 33, + 0xA03, 0xA03, 33, 0xA3E, 0xA40, 33, 0xA83, 0xA83, 33, 0xABE, 0xAC0, 33, + 0xAC9, 0xAC9, 33, 0xACB, 0xACC, 33, 0xB02, 0xB03, 33, 0xB3E, 0xB3E, 33, + 0xB40, 0xB40, 33, 0xB47, 0xB48, 33, 0xB4B, 0xB4C, 33, 0xB57, 0xB57, 33, + 0xBBE, 0xBBF, 33, 0xBC1, 0xBC2, 33, 0xBC6, 0xBC8, 33, 0xBCA, 0xBCC, 33, + 0xBD7, 0xBD7, 33, 0xC01, 0xC03, 33, 0xC41, 0xC44, 33, 0xC82, 0xC83, 33, + 0xCBE, 0xCBE, 33, 0xCC0, 0xCC4, 33, 0xCC7, 0xCC8, 33, 0xCCA, 0xCCB, 33, + 0xCD5, 0xCD6, 33, 0xCF3, 0xCF3, 33, 0xD02, 0xD03, 33, 0xD3E, 0xD40, 33, + 0xD46, 0xD48, 33, 0xD4A, 0xD4C, 33, 0xD57, 0xD57, 33, 0xD82, 0xD83, 33, + 0xDCF, 0xDD1, 33, 0xDD8, 0xDDF, 33, 0xDF2, 0xDF3, 33, 0xF3E, 0xF3F, 33, + 0xF7F, 0xF7F, 33, 0x102B, 0x102C, 33, 0x1031, 0x1031, 33, 0x1038, 0x1038, 33, + 0x103B, 0x103C, 33, 0x1056, 0x1057, 33, 0x1062, 0x1064, 33, 0x1067, 0x106D, 33, + 0x1083, 0x1084, 33, 0x1087, 0x108C, 33, 0x108F, 0x108F, 33, 0x109A, 0x109C, 33, + 0x1715, 0x1715, 33, 0x1734, 0x1734, 33, 0x17B6, 0x17B6, 33, 0x17BE, 0x17C5, 33, + 0x17C7, 0x17C8, 33, 0x1923, 0x1926, 33, 0x1929, 0x192B, 33, 0x1930, 0x1931, 33, + 0x1933, 0x1938, 33, 0x1A19, 0x1A1A, 33, 0x1A55, 0x1A55, 33, 0x1A57, 0x1A57, 33, + 0x1A61, 0x1A61, 33, 0x1A63, 0x1A64, 33, 0x1A6D, 0x1A72, 33, 0x1B04, 0x1B04, 33, + 0x1B35, 0x1B35, 33, 0x1B3B, 0x1B3B, 33, 0x1B3D, 0x1B41, 33, 0x1B43, 0x1B44, 33, + 0x1B82, 0x1B82, 33, 0x1BA1, 0x1BA1, 33, 0x1BA6, 0x1BA7, 33, 0x1BAA, 0x1BAA, 33, + 0x1BE7, 0x1BE7, 33, 0x1BEA, 0x1BEC, 33, 0x1BEE, 0x1BEE, 33, 0x1BF2, 0x1BF3, 33, + 0x1C24, 0x1C2B, 33, 0x1C34, 0x1C35, 33, 0x1CE1, 0x1CE1, 33, 0x1CF7, 0x1CF7, 33, + 0x302E, 0x302F, 33, 0xA823, 0xA824, 33, 0xA827, 0xA827, 33, 0xA880, 0xA881, 33, + 0xA8B4, 0xA8C3, 33, 0xA952, 0xA953, 33, 0xA983, 0xA983, 33, 0xA9B4, 0xA9B5, 33, + 0xA9BA, 0xA9BB, 33, 0xA9BE, 0xA9C0, 33, 0xAA2F, 0xAA30, 33, 0xAA33, 0xAA34, 33, + 0xAA4D, 0xAA4D, 33, 0xAA7B, 0xAA7B, 33, 0xAA7D, 0xAA7D, 33, 0xAAEB, 0xAAEB, 33, + 0xAAEE, 0xAAEF, 33, 0xAAF5, 0xAAF5, 33, 0xABE3, 0xABE4, 33, 0xABE6, 0xABE7, 33, + 0xABE9, 0xABEA, 33, 0xABEC, 0xABEC, 33, 0x11000, 0x11000, 33, 0x11002, 0x11002, 33, + 0x11082, 0x11082, 33, 0x110B0, 0x110B2, 33, 0x110B7, 0x110B8, 33, 0x1112C, 0x1112C, 33, + 0x11145, 0x11146, 33, 0x11182, 0x11182, 33, 0x111B3, 0x111B5, 33, 0x111BF, 0x111C0, 33, + 0x111CE, 0x111CE, 33, 0x1122C, 0x1122E, 33, 0x11232, 0x11233, 33, 0x11235, 0x11235, 33, + 0x112E0, 0x112E2, 33, 0x11302, 0x11303, 33, 0x1133E, 0x1133F, 33, 0x11341, 0x11344, 33, + 0x11347, 0x11348, 33, 0x1134B, 0x1134D, 33, 0x11357, 0x11357, 33, 0x11362, 0x11363, 33, + 0x113B8, 0x113BA, 33, 0x113C2, 0x113C2, 33, 0x113C5, 0x113C5, 33, 0x113C7, 0x113CA, 33, + 0x113CC, 0x113CD, 33, 0x113CF, 0x113CF, 33, 0x11435, 0x11437, 33, 0x11440, 0x11441, 33, + 0x11445, 0x11445, 33, 0x114B0, 0x114B2, 33, 0x114B9, 0x114B9, 33, 0x114BB, 0x114BE, 33, + 0x114C1, 0x114C1, 33, 0x115AF, 0x115B1, 33, 0x115B8, 0x115BB, 33, 0x115BE, 0x115BE, 33, + 0x11630, 0x11632, 33, 0x1163B, 0x1163C, 33, 0x1163E, 0x1163E, 33, 0x116AC, 0x116AC, 33, + 0x116AE, 0x116AF, 33, 0x116B6, 0x116B6, 33, 0x1171E, 0x1171E, 33, 0x11720, 0x11721, 33, + 0x11726, 0x11726, 33, 0x1182C, 0x1182E, 33, 0x11838, 0x11838, 33, 0x11930, 0x11935, 33, + 0x11937, 0x11938, 33, 0x1193D, 0x1193D, 33, 0x11940, 0x11940, 33, 0x11942, 0x11942, 33, + 0x119D1, 0x119D3, 33, 0x119DC, 0x119DF, 33, 0x119E4, 0x119E4, 33, 0x11A39, 0x11A39, 33, + 0x11A57, 0x11A58, 33, 0x11A97, 0x11A97, 33, 0x11B61, 0x11B61, 33, 0x11B65, 0x11B65, 33, + 0x11B67, 0x11B67, 33, 0x11C2F, 0x11C2F, 33, 0x11C3E, 0x11C3E, 33, 0x11CA9, 0x11CA9, 33, + 0x11CB1, 0x11CB1, 33, 0x11CB4, 0x11CB4, 33, 0x11D8A, 0x11D8E, 33, 0x11D93, 0x11D94, 33, + 0x11D96, 0x11D96, 33, 0x11EF5, 0x11EF6, 33, 0x11F03, 0x11F03, 33, 0x11F34, 0x11F35, 33, + 0x11F3E, 0x11F3F, 33, 0x11F41, 0x11F41, 33, 0x1612A, 0x1612C, 33, 0x16F51, 0x16F87, 33, + 0x16FF0, 0x16FF1, 33, 0x1D165, 0x1D166, 33, 0x1D16D, 0x1D172, 33, 0x30, 0x39, 34, + 0x660, 0x669, 34, 0x6F0, 0x6F9, 34, 0x7C0, 0x7C9, 34, 0x966, 0x96F, 34, + 0x9E6, 0x9EF, 34, 0xA66, 0xA6F, 34, 0xAE6, 0xAEF, 34, 0xB66, 0xB6F, 34, + 0xBE6, 0xBEF, 34, 0xC66, 0xC6F, 34, 0xCE6, 0xCEF, 34, 0xD66, 0xD6F, 34, + 0xDE6, 0xDEF, 34, 0xE50, 0xE59, 34, 0xED0, 0xED9, 34, 0xF20, 0xF29, 34, + 0x1040, 0x1049, 34, 0x1090, 0x1099, 34, 0x17E0, 0x17E9, 34, 0x1810, 0x1819, 34, + 0x1946, 0x194F, 34, 0x19D0, 0x19D9, 34, 0x1A80, 0x1A89, 34, 0x1A90, 0x1A99, 34, + 0x1B50, 0x1B59, 34, 0x1BB0, 0x1BB9, 34, 0x1C40, 0x1C49, 34, 0x1C50, 0x1C59, 34, + 0xA620, 0xA629, 34, 0xA8D0, 0xA8D9, 34, 0xA900, 0xA909, 34, 0xA9D0, 0xA9D9, 34, + 0xA9F0, 0xA9F9, 34, 0xAA50, 0xAA59, 34, 0xABF0, 0xABF9, 34, 0xFF10, 0xFF19, 34, + 0x104A0, 0x104A9, 34, 0x10D30, 0x10D39, 34, 0x10D40, 0x10D49, 34, 0x11066, 0x1106F, 34, + 0x110F0, 0x110F9, 34, 0x11136, 0x1113F, 34, 0x111D0, 0x111D9, 34, 0x112F0, 0x112F9, 34, + 0x11450, 0x11459, 34, 0x114D0, 0x114D9, 34, 0x11650, 0x11659, 34, 0x116C0, 0x116C9, 34, + 0x116D0, 0x116E3, 34, 0x11730, 0x11739, 34, 0x118E0, 0x118E9, 34, 0x11950, 0x11959, 34, + 0x11BF0, 0x11BF9, 34, 0x11C50, 0x11C59, 34, 0x11D50, 0x11D59, 34, 0x11DA0, 0x11DA9, 34, + }; + } + + private static int[] rangeChunk22() { + return new int[] { + 0x11DE0, 0x11DE9, 34, 0x11F50, 0x11F59, 34, 0x16130, 0x16139, 34, 0x16A60, 0x16A69, 34, + 0x16AC0, 0x16AC9, 34, 0x16B50, 0x16B59, 34, 0x16D70, 0x16D79, 34, 0x1CCF0, 0x1CCF9, 34, + 0x1D7CE, 0x1D7FF, 34, 0x1E140, 0x1E149, 34, 0x1E2F0, 0x1E2F9, 34, 0x1E4F0, 0x1E4F9, 34, + 0x1E5F1, 0x1E5FA, 34, 0x1E950, 0x1E959, 34, 0x1FBF0, 0x1FBF9, 34, 0x16EE, 0x16F0, 34, + 0x2160, 0x2182, 34, 0x2185, 0x2188, 34, 0x3007, 0x3007, 34, 0x3021, 0x3029, 34, + 0x3038, 0x303A, 34, 0xA6E6, 0xA6EF, 34, 0x10140, 0x10174, 34, 0x10341, 0x10341, 34, + 0x1034A, 0x1034A, 34, 0x103D1, 0x103D5, 34, 0x12400, 0x1246E, 34, 0x16FF4, 0x16FF6, 34, + 0xB2, 0xB3, 34, 0xB9, 0xB9, 34, 0xBC, 0xBE, 34, 0x9F4, 0x9F9, 34, + 0xB72, 0xB77, 34, 0xBF0, 0xBF2, 34, 0xC78, 0xC7E, 34, 0xD58, 0xD5E, 34, + 0xD70, 0xD78, 34, 0xF2A, 0xF33, 34, 0x1369, 0x137C, 34, 0x17F0, 0x17F9, 34, + 0x19DA, 0x19DA, 34, 0x2070, 0x2070, 34, 0x2074, 0x2079, 34, 0x2080, 0x2089, 34, + 0x2150, 0x215F, 34, 0x2189, 0x2189, 34, 0x2460, 0x249B, 34, 0x24EA, 0x24FF, 34, + 0x2776, 0x2793, 34, 0x2CFD, 0x2CFD, 34, 0x3192, 0x3195, 34, 0x3220, 0x3229, 34, + 0x3248, 0x324F, 34, 0x3251, 0x325F, 34, 0x3280, 0x3289, 34, 0x32B1, 0x32BF, 34, + 0xA830, 0xA835, 34, 0x10107, 0x10133, 34, 0x10175, 0x10178, 34, 0x1018A, 0x1018B, 34, + 0x102E1, 0x102FB, 34, 0x10320, 0x10323, 34, 0x10858, 0x1085F, 34, 0x10879, 0x1087F, 34, + 0x108A7, 0x108AF, 34, 0x108FB, 0x108FF, 34, 0x10916, 0x1091B, 34, 0x109BC, 0x109BD, 34, + 0x109C0, 0x109CF, 34, 0x109D2, 0x109FF, 34, 0x10A40, 0x10A48, 34, 0x10A7D, 0x10A7E, 34, + 0x10A9D, 0x10A9F, 34, 0x10AEB, 0x10AEF, 34, 0x10B58, 0x10B5F, 34, 0x10B78, 0x10B7F, 34, + 0x10BA9, 0x10BAF, 34, 0x10CFA, 0x10CFF, 34, 0x10E60, 0x10E7E, 34, 0x10F1D, 0x10F26, 34, + 0x10F51, 0x10F54, 34, 0x10FC5, 0x10FCB, 34, 0x11052, 0x11065, 34, 0x111E1, 0x111F4, 34, + 0x1173A, 0x1173B, 34, 0x118EA, 0x118F2, 34, 0x11C5A, 0x11C6C, 34, 0x11FC0, 0x11FD4, 34, + 0x16B5B, 0x16B61, 34, 0x16E80, 0x16E96, 34, 0x1D2C0, 0x1D2D3, 34, 0x1D2E0, 0x1D2F3, 34, + 0x1D360, 0x1D378, 34, 0x1E8C7, 0x1E8CF, 34, 0x1EC71, 0x1ECAB, 34, 0x1ECAD, 0x1ECAF, 34, + 0x1ECB1, 0x1ECB4, 34, 0x1ED01, 0x1ED2D, 34, 0x1ED2F, 0x1ED3D, 34, 0x1F100, 0x1F10C, 34, + 0x2D, 0x2D, 35, 0x58A, 0x58A, 35, 0x5BE, 0x5BE, 35, 0x1400, 0x1400, 35, + 0x1806, 0x1806, 35, 0x2010, 0x2015, 35, 0x2E17, 0x2E17, 35, 0x2E1A, 0x2E1A, 35, + 0x2E3A, 0x2E3B, 35, 0x2E40, 0x2E40, 35, 0x2E5D, 0x2E5D, 35, 0x301C, 0x301C, 35, + 0x3030, 0x3030, 35, 0x30A0, 0x30A0, 35, 0xFE31, 0xFE32, 35, 0xFE58, 0xFE58, 35, + 0xFE63, 0xFE63, 35, 0xFF0D, 0xFF0D, 35, 0x10D6E, 0x10D6E, 35, 0x10EAD, 0x10EAD, 35, + 0x28, 0x28, 35, 0x5B, 0x5B, 35, 0x7B, 0x7B, 35, 0xF3A, 0xF3A, 35, + 0xF3C, 0xF3C, 35, 0x169B, 0x169B, 35, 0x201A, 0x201A, 35, 0x201E, 0x201E, 35, + 0x2045, 0x2045, 35, 0x207D, 0x207D, 35, 0x208D, 0x208D, 35, 0x2308, 0x2308, 35, + 0x230A, 0x230A, 35, 0x2329, 0x2329, 35, 0x2768, 0x2768, 35, 0x276A, 0x276A, 35, + 0x276C, 0x276C, 35, 0x276E, 0x276E, 35, 0x2770, 0x2770, 35, 0x2772, 0x2772, 35, + 0x2774, 0x2774, 35, 0x27C5, 0x27C5, 35, 0x27E6, 0x27E6, 35, 0x27E8, 0x27E8, 35, + 0x27EA, 0x27EA, 35, 0x27EC, 0x27EC, 35, 0x27EE, 0x27EE, 35, 0x2983, 0x2983, 35, + 0x2985, 0x2985, 35, 0x2987, 0x2987, 35, 0x2989, 0x2989, 35, 0x298B, 0x298B, 35, + 0x298D, 0x298D, 35, 0x298F, 0x298F, 35, 0x2991, 0x2991, 35, 0x2993, 0x2993, 35, + 0x2995, 0x2995, 35, 0x2997, 0x2997, 35, 0x29D8, 0x29D8, 35, 0x29DA, 0x29DA, 35, + 0x29FC, 0x29FC, 35, 0x2E22, 0x2E22, 35, 0x2E24, 0x2E24, 35, 0x2E26, 0x2E26, 35, + 0x2E28, 0x2E28, 35, 0x2E42, 0x2E42, 35, 0x2E55, 0x2E55, 35, 0x2E57, 0x2E57, 35, + 0x2E59, 0x2E59, 35, 0x2E5B, 0x2E5B, 35, 0x3008, 0x3008, 35, 0x300A, 0x300A, 35, + 0x300C, 0x300C, 35, 0x300E, 0x300E, 35, 0x3010, 0x3010, 35, 0x3014, 0x3014, 35, + 0x3016, 0x3016, 35, 0x3018, 0x3018, 35, 0x301A, 0x301A, 35, 0x301D, 0x301D, 35, + 0xFD3F, 0xFD3F, 35, 0xFE17, 0xFE17, 35, 0xFE35, 0xFE35, 35, 0xFE37, 0xFE37, 35, + 0xFE39, 0xFE39, 35, 0xFE3B, 0xFE3B, 35, 0xFE3D, 0xFE3D, 35, 0xFE3F, 0xFE3F, 35, + 0xFE41, 0xFE41, 35, 0xFE43, 0xFE43, 35, 0xFE47, 0xFE47, 35, 0xFE59, 0xFE59, 35, + 0xFE5B, 0xFE5B, 35, 0xFE5D, 0xFE5D, 35, 0xFF08, 0xFF08, 35, 0xFF3B, 0xFF3B, 35, + 0xFF5B, 0xFF5B, 35, 0xFF5F, 0xFF5F, 35, 0xFF62, 0xFF62, 35, 0x29, 0x29, 35, + 0x5D, 0x5D, 35, 0x7D, 0x7D, 35, 0xF3B, 0xF3B, 35, 0xF3D, 0xF3D, 35, + 0x169C, 0x169C, 35, 0x2046, 0x2046, 35, 0x207E, 0x207E, 35, 0x208E, 0x208E, 35, + 0x2309, 0x2309, 35, 0x230B, 0x230B, 35, 0x232A, 0x232A, 35, 0x2769, 0x2769, 35, + 0x276B, 0x276B, 35, 0x276D, 0x276D, 35, 0x276F, 0x276F, 35, 0x2771, 0x2771, 35, + 0x2773, 0x2773, 35, 0x2775, 0x2775, 35, 0x27C6, 0x27C6, 35, 0x27E7, 0x27E7, 35, + 0x27E9, 0x27E9, 35, 0x27EB, 0x27EB, 35, 0x27ED, 0x27ED, 35, 0x27EF, 0x27EF, 35, + 0x2984, 0x2984, 35, 0x2986, 0x2986, 35, 0x2988, 0x2988, 35, 0x298A, 0x298A, 35, + 0x298C, 0x298C, 35, 0x298E, 0x298E, 35, 0x2990, 0x2990, 35, 0x2992, 0x2992, 35, + 0x2994, 0x2994, 35, 0x2996, 0x2996, 35, 0x2998, 0x2998, 35, 0x29D9, 0x29D9, 35, + 0x29DB, 0x29DB, 35, 0x29FD, 0x29FD, 35, 0x2E23, 0x2E23, 35, 0x2E25, 0x2E25, 35, + 0x2E27, 0x2E27, 35, 0x2E29, 0x2E29, 35, 0x2E56, 0x2E56, 35, 0x2E58, 0x2E58, 35, + 0x2E5A, 0x2E5A, 35, 0x2E5C, 0x2E5C, 35, 0x3009, 0x3009, 35, 0x300B, 0x300B, 35, + 0x300D, 0x300D, 35, 0x300F, 0x300F, 35, 0x3011, 0x3011, 35, 0x3015, 0x3015, 35, + 0x3017, 0x3017, 35, 0x3019, 0x3019, 35, 0x301B, 0x301B, 35, 0x301E, 0x301F, 35, + 0xFD3E, 0xFD3E, 35, 0xFE18, 0xFE18, 35, 0xFE36, 0xFE36, 35, 0xFE38, 0xFE38, 35, + 0xFE3A, 0xFE3A, 35, 0xFE3C, 0xFE3C, 35, 0xFE3E, 0xFE3E, 35, 0xFE40, 0xFE40, 35, + 0xFE42, 0xFE42, 35, 0xFE44, 0xFE44, 35, 0xFE48, 0xFE48, 35, 0xFE5A, 0xFE5A, 35, + 0xFE5C, 0xFE5C, 35, 0xFE5E, 0xFE5E, 35, 0xFF09, 0xFF09, 35, 0xFF3D, 0xFF3D, 35, + 0xFF5D, 0xFF5D, 35, 0xFF60, 0xFF60, 35, 0xFF63, 0xFF63, 35, 0x5F, 0x5F, 35, + 0x203F, 0x2040, 35, 0x2054, 0x2054, 35, 0xFE33, 0xFE34, 35, 0xFE4D, 0xFE4F, 35, + 0xFF3F, 0xFF3F, 35, 0x21, 0x23, 35, 0x25, 0x27, 35, 0x2A, 0x2A, 35, + 0x2C, 0x2C, 35, 0x2E, 0x2F, 35, 0x3A, 0x3B, 35, 0x3F, 0x40, 35, + 0x5C, 0x5C, 35, 0xA1, 0xA1, 35, 0xA7, 0xA7, 35, 0xB6, 0xB7, 35, + 0xBF, 0xBF, 35, 0x37E, 0x37E, 35, 0x387, 0x387, 35, 0x55A, 0x55F, 35, + 0x589, 0x589, 35, 0x5C0, 0x5C0, 35, 0x5C3, 0x5C3, 35, 0x5C6, 0x5C6, 35, + 0x5F3, 0x5F4, 35, 0x609, 0x60A, 35, 0x60C, 0x60D, 35, 0x61B, 0x61B, 35, + 0x61D, 0x61F, 35, 0x66A, 0x66D, 35, 0x6D4, 0x6D4, 35, 0x700, 0x70D, 35, + 0x7F7, 0x7F9, 35, 0x830, 0x83E, 35, 0x85E, 0x85E, 35, 0x964, 0x965, 35, + 0x970, 0x970, 35, 0x9FD, 0x9FD, 35, 0xA76, 0xA76, 35, 0xAF0, 0xAF0, 35, + 0xC77, 0xC77, 35, 0xC84, 0xC84, 35, 0xDF4, 0xDF4, 35, 0xE4F, 0xE4F, 35, + 0xE5A, 0xE5B, 35, 0xF04, 0xF12, 35, 0xF14, 0xF14, 35, 0xF85, 0xF85, 35, + 0xFD0, 0xFD4, 35, 0xFD9, 0xFDA, 35, 0x104A, 0x104F, 35, 0x10FB, 0x10FB, 35, + 0x1360, 0x1368, 35, 0x166E, 0x166E, 35, 0x16EB, 0x16ED, 35, 0x1735, 0x1736, 35, + 0x17D4, 0x17D6, 35, 0x17D8, 0x17DA, 35, 0x1800, 0x1805, 35, 0x1807, 0x180A, 35, + 0x1944, 0x1945, 35, 0x1A1E, 0x1A1F, 35, 0x1AA0, 0x1AA6, 35, 0x1AA8, 0x1AAD, 35, + 0x1B4E, 0x1B4F, 35, 0x1B5A, 0x1B60, 35, 0x1B7D, 0x1B7F, 35, 0x1BFC, 0x1BFF, 35, + 0x1C3B, 0x1C3F, 35, 0x1C7E, 0x1C7F, 35, 0x1CC0, 0x1CC7, 35, 0x1CD3, 0x1CD3, 35, + 0x2016, 0x2017, 35, 0x2020, 0x2027, 35, 0x2030, 0x2038, 35, 0x203B, 0x203E, 35, + 0x2041, 0x2043, 35, 0x2047, 0x2051, 35, 0x2053, 0x2053, 35, 0x2055, 0x205E, 35, + 0x2CF9, 0x2CFC, 35, 0x2CFE, 0x2CFF, 35, 0x2D70, 0x2D70, 35, 0x2E00, 0x2E01, 35, + 0x2E06, 0x2E08, 35, 0x2E0B, 0x2E0B, 35, 0x2E0E, 0x2E16, 35, 0x2E18, 0x2E19, 35, + 0x2E1B, 0x2E1B, 35, 0x2E1E, 0x2E1F, 35, 0x2E2A, 0x2E2E, 35, 0x2E30, 0x2E39, 35, + 0x2E3C, 0x2E3F, 35, 0x2E41, 0x2E41, 35, 0x2E43, 0x2E4F, 35, 0x2E52, 0x2E54, 35, + 0x3001, 0x3003, 35, 0x303D, 0x303D, 35, 0x30FB, 0x30FB, 35, 0xA4FE, 0xA4FF, 35, + 0xA60D, 0xA60F, 35, 0xA673, 0xA673, 35, 0xA67E, 0xA67E, 35, 0xA6F2, 0xA6F7, 35, + 0xA874, 0xA877, 35, 0xA8CE, 0xA8CF, 35, 0xA8F8, 0xA8FA, 35, 0xA8FC, 0xA8FC, 35, + 0xA92E, 0xA92F, 35, 0xA95F, 0xA95F, 35, 0xA9C1, 0xA9CD, 35, 0xA9DE, 0xA9DF, 35, + 0xAA5C, 0xAA5F, 35, 0xAADE, 0xAADF, 35, 0xAAF0, 0xAAF1, 35, 0xABEB, 0xABEB, 35, + 0xFE10, 0xFE16, 35, 0xFE19, 0xFE19, 35, 0xFE30, 0xFE30, 35, 0xFE45, 0xFE46, 35, + 0xFE49, 0xFE4C, 35, 0xFE50, 0xFE52, 35, 0xFE54, 0xFE57, 35, 0xFE5F, 0xFE61, 35, + }; + } + + private static int[] rangeChunk23() { + return new int[] { + 0xFE68, 0xFE68, 35, 0xFE6A, 0xFE6B, 35, 0xFF01, 0xFF03, 35, 0xFF05, 0xFF07, 35, + 0xFF0A, 0xFF0A, 35, 0xFF0C, 0xFF0C, 35, 0xFF0E, 0xFF0F, 35, 0xFF1A, 0xFF1B, 35, + 0xFF1F, 0xFF20, 35, 0xFF3C, 0xFF3C, 35, 0xFF61, 0xFF61, 35, 0xFF64, 0xFF65, 35, + 0x10100, 0x10102, 35, 0x1039F, 0x1039F, 35, 0x103D0, 0x103D0, 35, 0x1056F, 0x1056F, 35, + 0x10857, 0x10857, 35, 0x1091F, 0x1091F, 35, 0x1093F, 0x1093F, 35, 0x10A50, 0x10A58, 35, + 0x10A7F, 0x10A7F, 35, 0x10AF0, 0x10AF6, 35, 0x10B39, 0x10B3F, 35, 0x10B99, 0x10B9C, 35, + 0x10ED0, 0x10ED0, 35, 0x10F55, 0x10F59, 35, 0x10F86, 0x10F89, 35, 0x11047, 0x1104D, 35, + 0x110BB, 0x110BC, 35, 0x110BE, 0x110C1, 35, 0x11140, 0x11143, 35, 0x11174, 0x11175, 35, + 0x111C5, 0x111C8, 35, 0x111CD, 0x111CD, 35, 0x111DB, 0x111DB, 35, 0x111DD, 0x111DF, 35, + 0x11238, 0x1123D, 35, 0x112A9, 0x112A9, 35, 0x113D4, 0x113D5, 35, 0x113D7, 0x113D8, 35, + 0x1144B, 0x1144F, 35, 0x1145A, 0x1145B, 35, 0x1145D, 0x1145D, 35, 0x114C6, 0x114C6, 35, + 0x115C1, 0x115D7, 35, 0x11641, 0x11643, 35, 0x11660, 0x1166C, 35, 0x116B9, 0x116B9, 35, + 0x1173C, 0x1173E, 35, 0x1183B, 0x1183B, 35, 0x11944, 0x11946, 35, 0x119E2, 0x119E2, 35, + 0x11A3F, 0x11A46, 35, 0x11A9A, 0x11A9C, 35, 0x11A9E, 0x11AA2, 35, 0x11B00, 0x11B09, 35, + 0x11BE1, 0x11BE1, 35, 0x11C41, 0x11C45, 35, 0x11C70, 0x11C71, 35, 0x11EF7, 0x11EF8, 35, + 0x11F43, 0x11F4F, 35, 0x11FFF, 0x11FFF, 35, 0x12470, 0x12474, 35, 0x12FF1, 0x12FF2, 35, + 0x16A6E, 0x16A6F, 35, 0x16AF5, 0x16AF5, 35, 0x16B37, 0x16B3B, 35, 0x16B44, 0x16B44, 35, + 0x16D6D, 0x16D6F, 35, 0x16E97, 0x16E9A, 35, 0x16FE2, 0x16FE2, 35, 0x1BC9F, 0x1BC9F, 35, + 0x1DA87, 0x1DA8B, 35, 0x1E5FF, 0x1E5FF, 35, 0x1E95E, 0x1E95F, 35, 0xAB, 0xAB, 35, + 0x2018, 0x2018, 35, 0x201B, 0x201C, 35, 0x201F, 0x201F, 35, 0x2039, 0x2039, 35, + 0x2E02, 0x2E02, 35, 0x2E04, 0x2E04, 35, 0x2E09, 0x2E09, 35, 0x2E0C, 0x2E0C, 35, + 0x2E1C, 0x2E1C, 35, 0x2E20, 0x2E20, 35, 0xBB, 0xBB, 35, 0x2019, 0x2019, 35, + 0x201D, 0x201D, 35, 0x203A, 0x203A, 35, 0x2E03, 0x2E03, 35, 0x2E05, 0x2E05, 35, + 0x2E0A, 0x2E0A, 35, 0x2E0D, 0x2E0D, 35, 0x2E1D, 0x2E1D, 35, 0x2E21, 0x2E21, 35, + 0x2B, 0x2B, 36, 0x3C, 0x3E, 36, 0x7C, 0x7C, 36, 0x7E, 0x7E, 36, + 0xAC, 0xAC, 36, 0xB1, 0xB1, 36, 0xD7, 0xD7, 36, 0xF7, 0xF7, 36, + 0x3F6, 0x3F6, 36, 0x606, 0x608, 36, 0x2044, 0x2044, 36, 0x2052, 0x2052, 36, + 0x207A, 0x207C, 36, 0x208A, 0x208C, 36, 0x2118, 0x2118, 36, 0x2140, 0x2144, 36, + 0x214B, 0x214B, 36, 0x2190, 0x2194, 36, 0x219A, 0x219B, 36, 0x21A0, 0x21A0, 36, + 0x21A3, 0x21A3, 36, 0x21A6, 0x21A6, 36, 0x21AE, 0x21AE, 36, 0x21CE, 0x21CF, 36, + 0x21D2, 0x21D2, 36, 0x21D4, 0x21D4, 36, 0x21F4, 0x22FF, 36, 0x2320, 0x2321, 36, + 0x237C, 0x237C, 36, 0x239B, 0x23B3, 36, 0x23DC, 0x23E1, 36, 0x25B7, 0x25B7, 36, + 0x25C1, 0x25C1, 36, 0x25F8, 0x25FF, 36, 0x266F, 0x266F, 36, 0x27C0, 0x27C4, 36, + 0x27C7, 0x27E5, 36, 0x27F0, 0x27FF, 36, 0x2900, 0x2982, 36, 0x2999, 0x29D7, 36, + 0x29DC, 0x29FB, 36, 0x29FE, 0x2AFF, 36, 0x2B30, 0x2B44, 36, 0x2B47, 0x2B4C, 36, + 0xFB29, 0xFB29, 36, 0xFE62, 0xFE62, 36, 0xFE64, 0xFE66, 36, 0xFF0B, 0xFF0B, 36, + 0xFF1C, 0xFF1E, 36, 0xFF5C, 0xFF5C, 36, 0xFF5E, 0xFF5E, 36, 0xFFE2, 0xFFE2, 36, + 0xFFE9, 0xFFEC, 36, 0x10D8E, 0x10D8F, 36, 0x1CEF0, 0x1CEF0, 36, 0x1D6C1, 0x1D6C1, 36, + 0x1D6DB, 0x1D6DB, 36, 0x1D6FB, 0x1D6FB, 36, 0x1D715, 0x1D715, 36, 0x1D735, 0x1D735, 36, + 0x1D74F, 0x1D74F, 36, 0x1D76F, 0x1D76F, 36, 0x1D789, 0x1D789, 36, 0x1D7A9, 0x1D7A9, 36, + 0x1D7C3, 0x1D7C3, 36, 0x1EEF0, 0x1EEF1, 36, 0x1F8D0, 0x1F8D8, 36, 0x24, 0x24, 36, + 0xA2, 0xA5, 36, 0x58F, 0x58F, 36, 0x60B, 0x60B, 36, 0x7FE, 0x7FF, 36, + 0x9F2, 0x9F3, 36, 0x9FB, 0x9FB, 36, 0xAF1, 0xAF1, 36, 0xBF9, 0xBF9, 36, + 0xE3F, 0xE3F, 36, 0x17DB, 0x17DB, 36, 0x20A0, 0x20C1, 36, 0xA838, 0xA838, 36, + 0xFDFC, 0xFDFC, 36, 0xFE69, 0xFE69, 36, 0xFF04, 0xFF04, 36, 0xFFE0, 0xFFE1, 36, + 0xFFE5, 0xFFE6, 36, 0x11FDD, 0x11FE0, 36, 0x1E2FF, 0x1E2FF, 36, 0x1ECB0, 0x1ECB0, 36, + 0x5E, 0x5E, 36, 0x60, 0x60, 36, 0xA8, 0xA8, 36, 0xAF, 0xAF, 36, + 0xB4, 0xB4, 36, 0xB8, 0xB8, 36, 0x2C2, 0x2C5, 36, 0x2D2, 0x2DF, 36, + 0x2E5, 0x2EB, 36, 0x2ED, 0x2ED, 36, 0x2EF, 0x2FF, 36, 0x375, 0x375, 36, + 0x384, 0x385, 36, 0x888, 0x888, 36, 0x1FBD, 0x1FBD, 36, 0x1FBF, 0x1FC1, 36, + 0x1FCD, 0x1FCF, 36, 0x1FDD, 0x1FDF, 36, 0x1FED, 0x1FEF, 36, 0x1FFD, 0x1FFE, 36, + 0x309B, 0x309C, 36, 0xA700, 0xA716, 36, 0xA720, 0xA721, 36, 0xA789, 0xA78A, 36, + 0xAB5B, 0xAB5B, 36, 0xAB6A, 0xAB6B, 36, 0xFBB2, 0xFBC2, 36, 0xFF3E, 0xFF3E, 36, + 0xFF40, 0xFF40, 36, 0xFFE3, 0xFFE3, 36, 0x1F3FB, 0x1F3FF, 36, 0xA6, 0xA6, 36, + 0xA9, 0xA9, 36, 0xAE, 0xAE, 36, 0xB0, 0xB0, 36, 0x482, 0x482, 36, + 0x58D, 0x58E, 36, 0x60E, 0x60F, 36, 0x6DE, 0x6DE, 36, 0x6E9, 0x6E9, 36, + 0x6FD, 0x6FE, 36, 0x7F6, 0x7F6, 36, 0x9FA, 0x9FA, 36, 0xB70, 0xB70, 36, + 0xBF3, 0xBF8, 36, 0xBFA, 0xBFA, 36, 0xC7F, 0xC7F, 36, 0xD4F, 0xD4F, 36, + 0xD79, 0xD79, 36, 0xF01, 0xF03, 36, 0xF13, 0xF13, 36, 0xF15, 0xF17, 36, + 0xF1A, 0xF1F, 36, 0xF34, 0xF34, 36, 0xF36, 0xF36, 36, 0xF38, 0xF38, 36, + 0xFBE, 0xFC5, 36, 0xFC7, 0xFCC, 36, 0xFCE, 0xFCF, 36, 0xFD5, 0xFD8, 36, + 0x109E, 0x109F, 36, 0x1390, 0x1399, 36, 0x166D, 0x166D, 36, 0x1940, 0x1940, 36, + 0x19DE, 0x19FF, 36, 0x1B61, 0x1B6A, 36, 0x1B74, 0x1B7C, 36, 0x2100, 0x2101, 36, + 0x2103, 0x2106, 36, 0x2108, 0x2109, 36, 0x2114, 0x2114, 36, 0x2116, 0x2117, 36, + 0x211E, 0x2123, 36, 0x2125, 0x2125, 36, 0x2127, 0x2127, 36, 0x2129, 0x2129, 36, + 0x212E, 0x212E, 36, 0x213A, 0x213B, 36, 0x214A, 0x214A, 36, 0x214C, 0x214D, 36, + 0x214F, 0x214F, 36, 0x218A, 0x218B, 36, 0x2195, 0x2199, 36, 0x219C, 0x219F, 36, + 0x21A1, 0x21A2, 36, 0x21A4, 0x21A5, 36, 0x21A7, 0x21AD, 36, 0x21AF, 0x21CD, 36, + 0x21D0, 0x21D1, 36, 0x21D3, 0x21D3, 36, 0x21D5, 0x21F3, 36, 0x2300, 0x2307, 36, + 0x230C, 0x231F, 36, 0x2322, 0x2328, 36, 0x232B, 0x237B, 36, 0x237D, 0x239A, 36, + 0x23B4, 0x23DB, 36, 0x23E2, 0x2429, 36, 0x2440, 0x244A, 36, 0x249C, 0x24E9, 36, + 0x2500, 0x25B6, 36, 0x25B8, 0x25C0, 36, 0x25C2, 0x25F7, 36, 0x2600, 0x266E, 36, + 0x2670, 0x2767, 36, 0x2794, 0x27BF, 36, 0x2800, 0x28FF, 36, 0x2B00, 0x2B2F, 36, + 0x2B45, 0x2B46, 36, 0x2B4D, 0x2B73, 36, 0x2B76, 0x2BFF, 36, 0x2CE5, 0x2CEA, 36, + 0x2E50, 0x2E51, 36, 0x2E80, 0x2E99, 36, 0x2E9B, 0x2EF3, 36, 0x2F00, 0x2FD5, 36, + 0x2FF0, 0x2FFF, 36, 0x3004, 0x3004, 36, 0x3012, 0x3013, 36, 0x3020, 0x3020, 36, + 0x3036, 0x3037, 36, 0x303E, 0x303F, 36, 0x3190, 0x3191, 36, 0x3196, 0x319F, 36, + 0x31C0, 0x31E5, 36, 0x31EF, 0x31EF, 36, 0x3200, 0x321E, 36, 0x322A, 0x3247, 36, + 0x3250, 0x3250, 36, 0x3260, 0x327F, 36, 0x328A, 0x32B0, 36, 0x32C0, 0x33FF, 36, + 0x4DC0, 0x4DFF, 36, 0xA490, 0xA4C6, 36, 0xA828, 0xA82B, 36, 0xA836, 0xA837, 36, + 0xA839, 0xA839, 36, 0xAA77, 0xAA79, 36, 0xFBC3, 0xFBD2, 36, 0xFD40, 0xFD4F, 36, + 0xFD90, 0xFD91, 36, 0xFDC8, 0xFDCF, 36, 0xFDFD, 0xFDFF, 36, 0xFFE4, 0xFFE4, 36, + 0xFFE8, 0xFFE8, 36, 0xFFED, 0xFFEE, 36, 0xFFFC, 0xFFFD, 36, 0x10137, 0x1013F, 36, + 0x10179, 0x10189, 36, 0x1018C, 0x1018E, 36, 0x10190, 0x1019C, 36, 0x101A0, 0x101A0, 36, + 0x101D0, 0x101FC, 36, 0x10877, 0x10878, 36, 0x10AC8, 0x10AC8, 36, 0x10ED1, 0x10ED8, 36, + 0x1173F, 0x1173F, 36, 0x11FD5, 0x11FDC, 36, 0x11FE1, 0x11FF1, 36, 0x16B3C, 0x16B3F, 36, + 0x16B45, 0x16B45, 36, 0x1BC9C, 0x1BC9C, 36, 0x1CC00, 0x1CCEF, 36, 0x1CCFA, 0x1CCFC, 36, + 0x1CD00, 0x1CEB3, 36, 0x1CEBA, 0x1CED0, 36, 0x1CEE0, 0x1CEEF, 36, 0x1CF50, 0x1CFC3, 36, + 0x1D000, 0x1D0F5, 36, 0x1D100, 0x1D126, 36, 0x1D129, 0x1D164, 36, 0x1D16A, 0x1D16C, 36, + 0x1D183, 0x1D184, 36, 0x1D18C, 0x1D1A9, 36, 0x1D1AE, 0x1D1EA, 36, 0x1D200, 0x1D241, 36, + 0x1D245, 0x1D245, 36, 0x1D300, 0x1D356, 36, 0x1D800, 0x1D9FF, 36, 0x1DA37, 0x1DA3A, 36, + 0x1DA6D, 0x1DA74, 36, 0x1DA76, 0x1DA83, 36, 0x1DA85, 0x1DA86, 36, 0x1E14F, 0x1E14F, 36, + 0x1ECAC, 0x1ECAC, 36, 0x1ED2E, 0x1ED2E, 36, 0x1F000, 0x1F02B, 36, 0x1F030, 0x1F093, 36, + 0x1F0A0, 0x1F0AE, 36, 0x1F0B1, 0x1F0BF, 36, 0x1F0C1, 0x1F0CF, 36, 0x1F0D1, 0x1F0F5, 36, + 0x1F10D, 0x1F1AD, 36, 0x1F1E6, 0x1F202, 36, 0x1F210, 0x1F23B, 36, 0x1F240, 0x1F248, 36, + 0x1F250, 0x1F251, 36, 0x1F260, 0x1F265, 36, 0x1F300, 0x1F3FA, 36, 0x1F400, 0x1F6D8, 36, + 0x1F6DC, 0x1F6EC, 36, 0x1F6F0, 0x1F6FC, 36, 0x1F700, 0x1F7D9, 36, 0x1F7E0, 0x1F7EB, 36, + 0x1F7F0, 0x1F7F0, 36, 0x1F800, 0x1F80B, 36, 0x1F810, 0x1F847, 36, 0x1F850, 0x1F859, 36, + 0x1F860, 0x1F887, 36, 0x1F890, 0x1F8AD, 36, 0x1F8B0, 0x1F8BB, 36, 0x1F8C0, 0x1F8C1, 36, + 0x1F900, 0x1FA57, 36, 0x1FA60, 0x1FA6D, 36, 0x1FA70, 0x1FA7C, 36, 0x1FA80, 0x1FA8A, 36, + }; + } + + private static int[] rangeChunk24() { + return new int[] { + 0x1FA8E, 0x1FAC6, 36, 0x1FAC8, 0x1FAC8, 36, 0x1FACD, 0x1FADC, 36, 0x1FADF, 0x1FAEA, 36, + 0x1FAEF, 0x1FAF8, 36, 0x1FB00, 0x1FB92, 36, 0x1FB94, 0x1FBEF, 36, 0x1FBFA, 0x1FBFA, 36, + 0x20, 0x20, 37, 0xA0, 0xA0, 37, 0x1680, 0x1680, 37, 0x2000, 0x200A, 37, + 0x202F, 0x202F, 37, 0x205F, 0x205F, 37, 0x3000, 0x3000, 37, 0x2028, 0x2028, 37, + 0x2029, 0x2029, 37, + }; + } + + private static final int[][] RANGE_CHUNKS = { + rangeChunk0(), rangeChunk1(), rangeChunk2(), rangeChunk3(), rangeChunk4(), rangeChunk5(), rangeChunk6(), rangeChunk7(), rangeChunk8(), rangeChunk9(), rangeChunk10(), rangeChunk11(), rangeChunk12(), rangeChunk13(), rangeChunk14(), rangeChunk15(), rangeChunk16(), rangeChunk17(), rangeChunk18(), rangeChunk19(), rangeChunk20(), rangeChunk21(), rangeChunk22(), rangeChunk23(), rangeChunk24() + }; + + private static final String[] ALIASES = { + "c", "C", "casedletter", "LC", "cc", "Cc", "cf", "Cf", + "closepunctuation", "Pe", "cn", "Cn", "cntrl", "Cc", "co", "Co", + "combiningmark", "M", "connectorpunctuation", "Pc", "control", "Cc", "cs", "Cs", + "currencysymbol", "Sc", "dashpunctuation", "Pd", "decimalnumber", "Nd", "digit", "Nd", + "enclosingmark", "Me", "finalpunctuation", "Pf", "format", "Cf", "initialpunctuation", "Pi", + "l", "L", "lc", "LC", "letter", "L", "letternumber", "Nl", + "lineseparator", "Zl", "ll", "Ll", "lm", "Lm", "lo", "Lo", + "lowercaseletter", "Ll", "lt", "Lt", "lu", "Lu", "m", "M", + "mark", "M", "mathsymbol", "Sm", "mc", "Mc", "me", "Me", + "mn", "Mn", "modifierletter", "Lm", "modifiersymbol", "Sk", "n", "N", + "nd", "Nd", "nl", "Nl", "no", "No", "nonspacingmark", "Mn", + "number", "N", "openpunctuation", "Ps", "other", "C", "otherletter", "Lo", + "othernumber", "No", "otherpunctuation", "Po", "othersymbol", "So", "p", "P", + "paragraphseparator", "Zp", "pc", "Pc", "pd", "Pd", "pe", "Pe", + "pf", "Pf", "pi", "Pi", "po", "Po", "privateuse", "Co", + "ps", "Ps", "punct", "P", "punctuation", "P", "s", "S", + "sc", "Sc", "separator", "Z", "sk", "Sk", "sm", "Sm", + "so", "So", "spaceseparator", "Zs", "spacingmark", "Mc", "surrogate", "Cs", + "symbol", "S", "titlecaseletter", "Lt", "unassigned", "Cn", "uppercaseletter", "Lu", + "z", "Z", "zl", "Zl", "zp", "Zp", "zs", "Zs", + }; + + private static final UnicodeSet[] SETS = buildSets(); + private static final Map ALIAS_INDEX = buildAliasIndex(); + + static UnicodeSet resolve(String value) { + Integer index = ALIAS_INDEX.get(normalizeValue(value)); + return index == null ? null : SETS[index]; + } + + private static UnicodeSet[] buildSets() { + UnicodeSet[] sets = new UnicodeSet[VALUES.length]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + for (int[] ranges : RANGE_CHUNKS) { + for (int i = 0; i < ranges.length; i += 3) { + sets[ranges[i + 2]].add(ranges[i], ranges[i + 1]); + } + } + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + + private static Map buildAliasIndex() { + Map valueIndexes = new HashMap<>(); + for (int i = 0; i < VALUES.length; i++) valueIndexes.put(VALUES[i], i); + Map indexes = new HashMap<>(); + for (int i = 0; i < ALIASES.length; i += 2) { + indexes.put(ALIASES[i], valueIndexes.get(ALIASES[i + 1])); + } + return Map.copyOf(indexes); + } + + private static String normalizeValue(String value) { + String normalized = value.trim(); + if (normalized.startsWith(":\\A") && normalized.endsWith("\\z:") + && normalized.length() > 6) { + normalized = normalized.substring(3, normalized.length() - 3); + } + StringBuilder loose = new StringBuilder(normalized.length()); + for (int i = 0; i < normalized.length(); i++) { + char ch = normalized.charAt(i); + if (!Character.isWhitespace(ch) && ch != '-' && ch != '_') { + loose.append(Character.toLowerCase(ch)); + } + } + return loose.toString(); + } + + private PerlUnicodeGeneralCategoryData() { + } +} diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index 727f1556cb..4823ebbf17 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -936,6 +936,34 @@ private static UnicodeSet resolvePerlBuiltInPropertyAlias(String property) { if (property == null) return null; String alias = property.trim(); + int assignment = propertyValueDelimiter(alias); + if (assignment > 0 && assignment < alias.length() - 1 + && isGeneralCategoryProperty(alias.substring(0, assignment))) { + UnicodeSet category = PerlUnicodeGeneralCategoryData.resolve( + alias.substring(assignment + 1)); + if (category == null) { + throw new IllegalArgumentException( + "Unsupported General_Category value: " + + alias.substring(assignment + 1).trim()); + } + return category; + } + if (assignment > 0 && assignment < alias.length() - 1) { + Boolean value = perlBooleanPropertyValue(alias.substring(assignment + 1)); + if (value != null) { + UnicodeSet binaryProperty = resolvePerlBuiltInPropertyAlias( + alias.substring(0, assignment)); + if (binaryProperty == null + && loosePropertyName(alias.substring(0, assignment)) + .matches("(?:asciihexdigit|ahex)")) { + binaryProperty = new UnicodeSet() + .applyPropertyAlias("ASCII_Hex_Digit", "True"); + } + if (binaryProperty != null) { + return value ? binaryProperty : binaryProperty.complement(); + } + } + } if (alias.equalsIgnoreCase("L&")) { UnicodeSet casedLetters = unicodePropertyValueSet( UProperty.GENERAL_CATEGORY, "UppercaseLetter"); @@ -946,12 +974,6 @@ private static UnicodeSet resolvePerlBuiltInPropertyAlias(String property) { return casedLetters; } - int equals = alias.indexOf('='); - if (equals > 0 && loosePropertyName(alias.substring(0, equals)).equals("category")) { - return unicodePropertyValueSet( - UProperty.GENERAL_CATEGORY, alias.substring(equals + 1)); - } - String blockAlias = alias; if (alias.length() > 2 && alias.regionMatches(true, 0, "in", 0, 2)) { int valueStart = 2; @@ -962,7 +984,7 @@ private static UnicodeSet resolvePerlBuiltInPropertyAlias(String property) { } if (valueStart >= alias.length()) return null; blockAlias = alias.substring(valueStart); - } else if (equals >= 0 || unicodePropertyValue(UProperty.SCRIPT, alias) >= 0) { + } else if (assignment >= 0 || unicodePropertyValue(UProperty.SCRIPT, alias) >= 0) { return null; } @@ -977,6 +999,34 @@ private static UnicodeSet resolvePerlBuiltInPropertyAlias(String property) { return new UnicodeSet().applyIntPropertyValue(UProperty.BLOCK, blockValue); } + private static Boolean perlBooleanPropertyValue(String value) { + return switch (loosePropertyName(value)) { + case "true", "yes", "y", "t" -> true; + case "false", "no", "n", "f" -> false; + default -> null; + }; + } + + private static boolean isGeneralCategoryProperty(String property) { + return switch (loosePropertyName(property)) { + case "gc", "generalcategory", "category" -> true; + default -> false; + }; + } + + private static int propertyValueDelimiter(String property) { + int equals = property.indexOf('='); + if (equals >= 0) return equals; + for (int i = 1; i < property.length() - 1; i++) { + if (property.charAt(i) == ':' + && property.charAt(i - 1) != ':' + && property.charAt(i + 1) != ':') { + return i; + } + } + return -1; + } + private static String loosePropertyName(String value) { StringBuilder normalized = new StringBuilder(value.length()); for (int i = 0; i < value.length(); i++) { diff --git a/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeGeneralCategoryDataTest.java b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeGeneralCategoryDataTest.java new file mode 100644 index 0000000000..c7f81a09de --- /dev/null +++ b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeGeneralCategoryDataTest.java @@ -0,0 +1,25 @@ +package org.perlonjava.runtime.regex; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; + +@Tag("unit") +class PerlUnicodeGeneralCategoryDataTest { + @Test + void usesPinnedPerl544Unicode17DataAndAliases() { + assertEquals("17.0.0", PerlUnicodeGeneralCategoryData.UNICODE_VERSION); + assertTrue(PerlUnicodeGeneralCategoryData.resolve("uppercase_letter").contains('A')); + assertTrue(PerlUnicodeGeneralCategoryData.resolve(":\\ALu\\z:").contains('A')); + assertTrue(PerlUnicodeGeneralCategoryData.resolve("L").contains('A')); + assertTrue(PerlUnicodeGeneralCategoryData.resolve("LC").contains('a')); + assertTrue(PerlUnicodeGeneralCategoryData.resolve("Lo").contains(0x33479)); + assertTrue(PerlUnicodeGeneralCategoryData.resolve("Cn").contains(0x3347a)); + assertFalse(PerlUnicodeGeneralCategoryData.resolve("Cn").contains(0x33479)); + assertNull(PerlUnicodeGeneralCategoryData.resolve("not_a_category")); + } +} diff --git a/src/test/resources/unit/regex/line_boundary_rules.t b/src/test/resources/unit/regex/line_boundary_rules.t new file mode 100644 index 0000000000..28224ea6c4 --- /dev/null +++ b/src/test/resources/unit/regex/line_boundary_rules.t @@ -0,0 +1,71 @@ +use strict; +use warnings; +use utf8; +use Test::More tests => 84; + +sub has_boundary { + my ($left, $right) = @_; + my $text = $left . $right; + return $text =~ /\A\Q$left\E\b{lb}\Q$right\E\z/u; +} + +sub has_nonboundary { + my ($left, $right) = @_; + my $text = $left . $right; + return $text =~ /\A\Q$left\E\B{lb}\Q$right\E\z/u; +} + +sub boundary { + my ($left, $right, $rule) = @_; + ok(has_boundary($left, $right), "$rule permits a boundary"); + ok(!has_nonboundary($left, $right), "$rule rejects the inverse"); +} + +sub nonboundary { + my ($left, $right, $rule) = @_; + ok(!has_boundary($left, $right), "$rule rejects a boundary"); + ok(has_nonboundary($left, $right), "$rule accepts the inverse"); +} + +nonboundary('', 'A', 'LB2 start of text'); +boundary('A', '', 'LB3 end of text'); +boundary("\x{B}", 'A', 'LB4 mandatory break'); +nonboundary("\r", "\n", 'LB5 CR LF pair'); +boundary("\r", 'A', 'LB5 after CR'); +nonboundary('A', "\n", 'LB6 before hard break'); +nonboundary('A', ' ', 'LB7 before space'); +boundary("\x{200B} ", 'A', 'LB8 after zero-width space'); +nonboundary("\x{200D}", 'A', 'LB8a after ZWJ'); +nonboundary('A', "\x{0308}", 'LB9 combining sequence'); +boundary(' ', "\x{0308}", 'LB10 leading combining mark becomes AL'); +nonboundary('A', "\x{2060}", 'LB11 before word joiner'); +nonboundary("\x{2060}", 'A', 'LB11 after word joiner'); +nonboundary("\x{00A0}", 'A', 'LB12 after glue'); +nonboundary('A', "\x{00A0}", 'LB12a before glue'); +boundary(' ', "\x{00A0}", 'LB12a space overrides glue'); +nonboundary('A ', ']', 'LB13 before closing punctuation'); +nonboundary('( ', 'A', 'LB14 after opening punctuation'); +boundary(' ', '.5', 'LB15c space before infix-number sequence'); +nonboundary('A ', ',', 'LB15d before infix separator'); +nonboundary(') ', "\x{3041}", 'LB16 closing punctuation before nonstarter'); +nonboundary("\x{2014} ", "\x{2014}", 'LB17 break-both pair'); +boundary('A ', 'B', 'LB18 after space'); +nonboundary('A', '"', 'LB19 before unresolved quote'); +boundary('A', "\x{FFFC}", 'LB20 before contingent break'); +boundary("\x{FFFC}", 'A', 'LB20 after contingent break'); +nonboundary(' -', 'A', 'LB20a word-initial hyphen suppression'); +nonboundary('A', '-', 'LB21 before hyphen'); +nonboundary('A', "\x{2026}", 'LB22 before inseparable ellipsis'); +nonboundary('A', '1', 'LB23 letter digit'); +nonboundary('$', "\x{4E2D}", 'LB23a prefix ideograph'); +nonboundary('$', 'A', 'LB24 prefix alphabetic'); +nonboundary('12,', '3', 'LB25 numeric expression'); +nonboundary("\x{1100}", "\x{1161}", 'LB26 Hangul syllable'); +nonboundary('A', 'B', 'LB28 alphabetic word'); +nonboundary("\x{11003}", "\x{1B05}", 'LB28a Brahmic syllable'); +nonboundary('.', 'A', 'LB29 numeric punctuation alphabetic'); +nonboundary('A', '(', 'LB30 alphabetic before parenthesis'); +nonboundary("\x{1F1E6}", "\x{1F1E7}", 'LB30a odd regional-indicator prefix'); +boundary("\x{1F1E6}\x{1F1E7}", "\x{1F1E8}", 'LB30a even regional-indicator prefix'); +nonboundary("\x{261D}", "\x{1F3FB}", 'LB30b emoji base modifier'); +boundary("\x{4E2D}", "\x{6587}", 'LB31 default ideograph break'); diff --git a/src/test/resources/unit/regex/unicode_binary_property_values.t b/src/test/resources/unit/regex/unicode_binary_property_values.t new file mode 100644 index 0000000000..f6241e6a48 --- /dev/null +++ b/src/test/resources/unit/regex/unicode_binary_property_values.t @@ -0,0 +1,16 @@ +use strict; +use warnings; +use Test::More tests => 16; + +for my $pair ( + [ True => 1 ], [ False => 0 ], + [ Yes => 1 ], [ No => 0 ], + [ Y => 1 ], [ N => 0 ], + [ T => 1 ], [ F => 0 ], +) { + my ($value, $expected) = @$pair; + my $property = "ASCII_Hex_Digit=$value"; + my $positive = eval qq{'A' =~ /\\p{$property}/}; + is($@, '', "$property compiles"); + is(!!$positive, !!$expected, "$property has Perl boolean semantics"); +} diff --git a/src/test/resources/unit/regex/unicode_general_category_pinned.t b/src/test/resources/unit/regex/unicode_general_category_pinned.t new file mode 100644 index 0000000000..ded00b2094 --- /dev/null +++ b/src/test/resources/unit/regex/unicode_general_category_pinned.t @@ -0,0 +1,25 @@ +use strict; +use warnings; +use Test::More tests => 18; +no warnings 'experimental::uniprop_wildcards'; + +ok('A' =~ /\p{Gc=Lu}/, 'short property and value aliases'); +ok('A' =~ /\p{General_Category=Uppercase_Letter}/, 'long aliases'); +ok('A' =~ /\p{Category=uppercaseletter}/, 'Perl Category property alias'); +ok('A' =~ /\p{Gc: Lu}/, 'colon property delimiter'); +ok('A' =~ /\p{Is_General_Category=Lu}/, 'Is property prefix'); +ok('A' =~ /\p{Gc=:\ALu\z:}/, 'anchored wildcard value'); +ok('A' =~ /\p{General_Category= --Uppercase_letter}/, + 'loose hyphen, underscore, and whitespace matching'); +ok('A' =~ /\p{Gc=L}/, 'composite Letter category'); +ok('A' =~ /\p{Gc=LC}/, 'composite Cased_Letter category'); +ok('1' =~ /\p{Gc=N}/, 'composite Number category'); +ok('!' =~ /\p{Gc=P}/, 'composite Punctuation category'); +ok('$' =~ /\p{Gc=S}/, 'composite Symbol category'); +ok("\n" =~ /\p{Gc=C}/, 'composite Other category'); +ok('a' !~ /\p{Gc=Lu}/, 'exact category excludes other letters'); +ok('a' =~ /\P{Gc=Lu}/, 'outer property negation'); +ok('a' =~ /\p{^Gc=Lu}/, 'inner caret property negation'); +ok('A' =~ /\P{^Gc=Lu}/, 'double property negation'); +ok("\x{301}" =~ /\p{General_Category=Nonspacing_Mark}/, + 'nonspacing mark long value'); diff --git a/third_party/joni/src/org/joni/ArrayCompiler.java b/third_party/joni/src/org/joni/ArrayCompiler.java index d3579bf3e4..ef595fd1a6 100644 --- a/third_party/joni/src/org/joni/ArrayCompiler.java +++ b/third_party/joni/src/org/joni/ArrayCompiler.java @@ -1222,6 +1222,14 @@ protected void compileAnchorNode(AnchorNode node) { addOpcode(OPCode.NOT_WORD_BREAK_BOUNDARY); break; + case AnchorType.LINE_BOUNDARY: + addOpcode(OPCode.LINE_BOUNDARY); + break; + + case AnchorType.NOT_LINE_BOUNDARY: + addOpcode(OPCode.NOT_LINE_BOUNDARY); + break; + case AnchorType.WORD_BEGIN: if (Config.USE_WORD_BEGIN_END) { if (node.asciiRange) { diff --git a/third_party/joni/src/org/joni/ByteCodeMachine.java b/third_party/joni/src/org/joni/ByteCodeMachine.java index c918696193..2c7ffde711 100644 --- a/third_party/joni/src/org/joni/ByteCodeMachine.java +++ b/third_party/joni/src/org/joni/ByteCodeMachine.java @@ -256,6 +256,8 @@ private final int execute(final boolean checkThreadInterrupt) throws Interrupted case OPCode.NOT_SENTENCE_BOUNDARY: opSentenceBoundary(true); continue; case OPCode.WORD_BREAK_BOUNDARY: opWordBreakBoundary(false); continue; case OPCode.NOT_WORD_BREAK_BOUNDARY: opWordBreakBoundary(true); continue; + case OPCode.LINE_BOUNDARY: opLineBoundary(false); continue; + case OPCode.NOT_LINE_BOUNDARY: opLineBoundary(true); continue; case OPCode.WORD_BEGIN: opWordBegin(); continue; case OPCode.WORD_END: opWordEnd(); continue; @@ -419,6 +421,8 @@ private final int executeSb(final boolean checkThreadInterrupt) throws Interrupt case OPCode.NOT_SENTENCE_BOUNDARY: opSentenceBoundary(true); continue; case OPCode.WORD_BREAK_BOUNDARY: opWordBreakBoundary(false); continue; case OPCode.NOT_WORD_BREAK_BOUNDARY: opWordBreakBoundary(true); continue; + case OPCode.LINE_BOUNDARY: opLineBoundary(false); continue; + case OPCode.NOT_LINE_BOUNDARY: opLineBoundary(true); continue; case OPCode.WORD_BEGIN: opWordBeginSb(); continue; case OPCode.WORD_END: opWordEndSb(); continue; @@ -1387,6 +1391,341 @@ private boolean isWordExtendNumLetBase(byte property) { || property == WordBreakData.EXTEND_NUM_LET; } + private void opLineBoundary(boolean negated) { + if (isLineBoundary() == negated) opFail(); + } + + private boolean isLineBoundary() { + if (s <= str) return false; // LB2 + if (s >= end) return true; // LB3 + + int rawLeftPosition = enc.prevCharHead(bytes, str, s, end); + short rawLeft = rawLinePropertyAt(rawLeftPosition); + short rawRight = rawLinePropertyAt(s); + short rawLeftClass = lineClass(rawLeft); + short rawRightClass = lineClass(rawRight); + + if (rawLeftClass == LineBreakData.CR && rawRightClass == LineBreakData.LF) return false; // LB5 + if (isLineHardBreak(rawLeftClass)) return true; // LB4, LB5 + if (isLineHardBreak(rawRightClass)) return false; // LB6 + if (rawRightClass == LineBreakData.SP || rawRightClass == LineBreakData.ZW) return false; // LB7 + + int scan = rawLeftPosition; + while (scan >= str && lineClass(effectiveLinePropertyAt(scan)) == LineBreakData.SP) { + scan = previousRawLinePosition(scan); + } + if (scan >= str && lineClass(effectiveLinePropertyAt(scan)) == LineBreakData.ZW) return true; // LB8 + if (rawLeftClass == LineBreakData.ZWJ) return false; // LB8a + + if ((rawRightClass == LineBreakData.CM || rawRightClass == LineBreakData.ZWJ) + && !isLineCombiningException(lineClass(effectiveLinePropertyAt(rawLeftPosition)))) { + return false; // LB9 + } + + int leftPosition = previousLinePosition(s); + int rightPosition = nextLinePosition(s); + if (leftPosition < str || rightPosition >= end) return true; + short leftProperty = effectiveLinePropertyAt(leftPosition); + short rightProperty = effectiveLinePropertyAt(rightPosition); + short left = lineClass(leftProperty); + short right = lineClass(rightProperty); + + if (left == LineBreakData.WJ || right == LineBreakData.WJ) return false; // LB11 + if (left == LineBreakData.GL) return false; // LB12 + if (right == LineBreakData.GL + && left != LineBreakData.SP && left != LineBreakData.BA + && left != LineBreakData.HY && left != LineBreakData.HH) return false; // LB12a + if (right == LineBreakData.CL || right == LineBreakData.CP + || right == LineBreakData.EX || right == LineBreakData.SY) return false; // LB13 + if (lineOpenBeforeSpaces(leftPosition)) return false; // LB14 + if (lineInitialQuoteBeforeSpaces(leftPosition)) return false; // LB15a + if (lineFinalQuoteAt(rightPosition, rightProperty)) return false; // LB15b + if (left == LineBreakData.SP && right == LineBreakData.IS + && lineClassAt(nextLinePositionAfter(rightPosition)) == LineBreakData.NU) return true; // LB15c + if (right == LineBreakData.IS) return false; // LB15d + if (right == LineBreakData.NS && lineCloseBeforeSpaces(leftPosition)) return false; // LB16 + if (right == LineBreakData.B2 && lineB2BeforeSpaces(leftPosition)) return false; // LB17 + if (left == LineBreakData.SP) return true; // LB18 + if ((right == LineBreakData.QU && !hasLineFlag(rightProperty, LineBreakData.INITIAL_PUNCTUATION)) + || (left == LineBreakData.QU && !hasLineFlag(leftProperty, LineBreakData.FINAL_PUNCTUATION))) return false; // LB19 + if (lineQuoteEastAsianSuppression(leftPosition, rightPosition, leftProperty, rightProperty)) return false; // LB19a + if (left == LineBreakData.CB || right == LineBreakData.CB) return true; // LB20 + if (lineWordInitialHyphen(leftPosition, left, right)) return false; // LB20a + if (right == LineBreakData.BA || right == LineBreakData.HH + || right == LineBreakData.HY || right == LineBreakData.NS + || left == LineBreakData.BB) return false; // LB21 + if (lineHebrewHyphenSuppression(leftPosition, left, right)) return false; // LB21a + if (left == LineBreakData.SY && right == LineBreakData.HL) return false; // LB21b + if (right == LineBreakData.IN) return false; // LB22 + if ((isLineAlphabetic(left) && right == LineBreakData.NU) + || (left == LineBreakData.NU && isLineAlphabetic(right))) return false; // LB23 + if ((left == LineBreakData.PR && isLineIdeographicEmoji(right)) + || (isLineIdeographicEmoji(left) && right == LineBreakData.PO)) return false; // LB23a + if ((isLinePrefixPostfix(left) && isLineAlphabetic(right)) + || (isLineAlphabetic(left) && isLinePrefixPostfix(right))) return false; // LB24 + if (lineNumericSuppression(leftPosition, rightPosition, left, right)) return false; // LB25 + if (lineHangulSuppression(left, right)) return false; // LB26 + if ((isLineHangul(left) && right == LineBreakData.PO) + || (left == LineBreakData.PR && isLineHangul(right))) return false; // LB27 + if (isLineAlphabetic(left) && isLineAlphabetic(right)) return false; // LB28 + if (lineBrahmicSuppression(leftPosition, rightPosition, left, right)) return false; // LB28a + if (left == LineBreakData.IS && isLineAlphabetic(right)) return false; // LB29 + if ((isLineAlphaNumeric(left) && right == LineBreakData.OP + && !hasLineFlag(rightProperty, LineBreakData.EAST_ASIAN)) + || (left == LineBreakData.CP && !hasLineFlag(leftProperty, LineBreakData.EAST_ASIAN) + && isLineAlphaNumeric(right))) return false; // LB30 + if (left == LineBreakData.RI && right == LineBreakData.RI + && (precedingLineRun(leftPosition, LineBreakData.RI) & 1) == 1) return false; // LB30a + if ((left == LineBreakData.EB + || hasLineFlag(leftProperty, LineBreakData.UNASSIGNED_EXTENDED_PICTOGRAPHIC)) + && right == LineBreakData.EM) return false; // LB30b + return true; // LB31 + } + + private short rawLinePropertyAt(int position) { + return LineBreakData.propertyOf(enc.mbcToCode(bytes, position, end)); + } + + private short effectiveLinePropertyAt(int position) { + short property = rawLinePropertyAt(position); + short propertyClass = lineClass(property); + if (propertyClass != LineBreakData.CM && propertyClass != LineBreakData.ZWJ) return property; + + int cursor = position; + while (cursor > str) { + cursor = previousRawLinePosition(cursor); + short previous = rawLinePropertyAt(cursor); + short previousClass = lineClass(previous); + if (previousClass != LineBreakData.CM && previousClass != LineBreakData.ZWJ) { + return isLineCombiningException(previousClass) ? LineBreakData.AL : previous; + } + } + return LineBreakData.AL; + } + + private short lineClass(short property) { + return (short)(property & LineBreakData.CLASS_MASK); + } + + private short lineClassAt(int position) { + return position >= str && position < end + ? lineClass(effectiveLinePropertyAt(position)) : -1; + } + + private boolean hasLineFlag(short property, short flag) { + return (property & flag) != 0; + } + + private int previousRawLinePosition(int position) { + return position > str ? enc.prevCharHead(bytes, str, position, end) : -1; + } + + private int previousLinePosition(int position) { + int previous = previousRawLinePosition(position); + while (previous >= str && isIgnoredLinePosition(previous)) previous = previousRawLinePosition(previous); + return previous; + } + + private int nextLinePosition(int position) { + int next = position; + while (next < end && isIgnoredLinePosition(next)) next += enc.length(bytes, next, end); + return next; + } + + private int nextLinePositionAfter(int position) { + if (position < str || position >= end) return end; + return nextLinePosition(position + enc.length(bytes, position, end)); + } + + private boolean isIgnoredLinePosition(int position) { + short raw = lineClass(rawLinePropertyAt(position)); + if (raw != LineBreakData.CM && raw != LineBreakData.ZWJ) return false; + int previous = previousRawLinePosition(position); + return previous >= str && !isLineCombiningException(lineClass(effectiveLinePropertyAt(previous))); + } + + private boolean isLineCombiningException(short value) { + return value == LineBreakData.BK || value == LineBreakData.CR || value == LineBreakData.LF + || value == LineBreakData.NL || value == LineBreakData.SP || value == LineBreakData.ZW; + } + + private boolean isLineHardBreak(short value) { + return value == LineBreakData.BK || value == LineBreakData.CR + || value == LineBreakData.LF || value == LineBreakData.NL; + } + + private boolean isLineAlphabetic(short value) { + return value == LineBreakData.AL || value == LineBreakData.HL; + } + + private boolean isLineAlphaNumeric(short value) { + return isLineAlphabetic(value) || value == LineBreakData.NU; + } + + private boolean isLinePrefixPostfix(short value) { + return value == LineBreakData.PR || value == LineBreakData.PO; + } + + private boolean isLineIdeographicEmoji(short value) { + return value == LineBreakData.ID || value == LineBreakData.EB || value == LineBreakData.EM; + } + + private boolean isLineHangul(short value) { + return value == LineBreakData.JL || value == LineBreakData.JV || value == LineBreakData.JT + || value == LineBreakData.H2 || value == LineBreakData.H3; + } + + private boolean lineOpenBeforeSpaces(int leftPosition) { + int cursor = leftPosition; + while (cursor >= str && lineClassAt(cursor) == LineBreakData.SP) cursor = previousLinePosition(cursor); + return cursor >= str && lineClassAt(cursor) == LineBreakData.OP; + } + + private boolean lineInitialQuoteBeforeSpaces(int leftPosition) { + int cursor = leftPosition; + while (cursor >= str && lineClassAt(cursor) == LineBreakData.SP) cursor = previousLinePosition(cursor); + if (cursor < str) return false; + short quote = effectiveLinePropertyAt(cursor); + if (lineClass(quote) != LineBreakData.QU + || !hasLineFlag(quote, LineBreakData.INITIAL_PUNCTUATION)) return false; + cursor = previousLinePosition(cursor); + if (cursor < str) return true; + short value = lineClassAt(cursor); + return isLineHardBreak(value) || value == LineBreakData.OP || value == LineBreakData.QU + || value == LineBreakData.GL || value == LineBreakData.SP || value == LineBreakData.ZW; + } + + private boolean lineFinalQuoteAt(int rightPosition, short rightProperty) { + if (lineClass(rightProperty) != LineBreakData.QU + || !hasLineFlag(rightProperty, LineBreakData.FINAL_PUNCTUATION)) return false; + int next = nextLinePositionAfter(rightPosition); + if (next >= end) return true; + short value = lineClassAt(next); + return value == LineBreakData.SP || value == LineBreakData.GL || value == LineBreakData.WJ + || value == LineBreakData.CL || value == LineBreakData.QU || value == LineBreakData.CP + || value == LineBreakData.EX || value == LineBreakData.IS || value == LineBreakData.SY + || isLineHardBreak(value) || value == LineBreakData.ZW; + } + + private boolean lineCloseBeforeSpaces(int leftPosition) { + int cursor = leftPosition; + while (cursor >= str && lineClassAt(cursor) == LineBreakData.SP) cursor = previousLinePosition(cursor); + if (cursor < str) return false; + short value = lineClassAt(cursor); + return value == LineBreakData.CL || value == LineBreakData.CP; + } + + private boolean lineB2BeforeSpaces(int leftPosition) { + int cursor = leftPosition; + while (cursor >= str && lineClassAt(cursor) == LineBreakData.SP) cursor = previousLinePosition(cursor); + return cursor >= str && lineClassAt(cursor) == LineBreakData.B2; + } + + private boolean lineQuoteEastAsianSuppression(int leftPosition, int rightPosition, + short leftProperty, short rightProperty) { + if (lineClass(rightProperty) == LineBreakData.QU) { + if (!hasLineFlag(leftProperty, LineBreakData.EAST_ASIAN)) return true; + int next = nextLinePositionAfter(rightPosition); + if (next >= end || !hasLineFlag(effectiveLinePropertyAt(next), LineBreakData.EAST_ASIAN)) return true; + } + if (lineClass(leftProperty) == LineBreakData.QU) { + if (!hasLineFlag(rightProperty, LineBreakData.EAST_ASIAN)) return true; + int previous = previousLinePosition(leftPosition); + if (previous < str || !hasLineFlag(effectiveLinePropertyAt(previous), LineBreakData.EAST_ASIAN)) return true; + } + return false; + } + + private boolean lineWordInitialHyphen(int leftPosition, short left, short right) { + if (right != LineBreakData.AL && right != LineBreakData.HL) return false; + if (left != LineBreakData.HY && left != LineBreakData.HH) return false; + int previous = previousLinePosition(leftPosition); + if (previous < str) return true; + short value = lineClassAt(previous); + return isLineHardBreak(value) || value == LineBreakData.SP || value == LineBreakData.ZW + || value == LineBreakData.CB || value == LineBreakData.GL; + } + + private boolean lineHebrewHyphenSuppression(int leftPosition, short left, short right) { + if (right == LineBreakData.HL || (left != LineBreakData.HY && left != LineBreakData.HH)) return false; + int previous = previousLinePosition(leftPosition); + return previous >= str && lineClassAt(previous) == LineBreakData.HL; + } + + private boolean lineNumericSuppression(int leftPosition, int rightPosition, short left, short right) { + if ((left == LineBreakData.PR || left == LineBreakData.PO) + && (right == LineBreakData.NU || lineOpenStartsNumeric(rightPosition))) return true; + if ((left == LineBreakData.HY || left == LineBreakData.IS) && right == LineBreakData.NU) return true; + if (right == LineBreakData.NU) { + int cursor = leftPosition; + while (cursor >= str) { + short value = lineClassAt(cursor); + if (value != LineBreakData.SY && value != LineBreakData.IS) return value == LineBreakData.NU; + cursor = previousLinePosition(cursor); + } + } + if (right == LineBreakData.PR || right == LineBreakData.PO) { + int cursor = leftPosition; + if (lineClassAt(cursor) == LineBreakData.CL || lineClassAt(cursor) == LineBreakData.CP) { + cursor = previousLinePosition(cursor); + } + while (cursor >= str) { + short value = lineClassAt(cursor); + if (value != LineBreakData.SY && value != LineBreakData.IS) return value == LineBreakData.NU; + cursor = previousLinePosition(cursor); + } + } + return false; + } + + private boolean lineOpenStartsNumeric(int rightPosition) { + if (lineClassAt(rightPosition) != LineBreakData.OP) return false; + int next = nextLinePositionAfter(rightPosition); + if (lineClassAt(next) == LineBreakData.IS) next = nextLinePositionAfter(next); + return lineClassAt(next) == LineBreakData.NU; + } + + private boolean lineHangulSuppression(short left, short right) { + return (left == LineBreakData.JL && (right == LineBreakData.JL || right == LineBreakData.JV + || right == LineBreakData.H2 || right == LineBreakData.H3)) + || ((left == LineBreakData.JV || left == LineBreakData.H2) + && (right == LineBreakData.JV || right == LineBreakData.JT)) + || ((left == LineBreakData.JT || left == LineBreakData.H3) && right == LineBreakData.JT); + } + + private boolean lineBrahmicSuppression(int leftPosition, int rightPosition, short left, short right) { + boolean leftBase = left == LineBreakData.AK || left == LineBreakData.AS || lineCodePointAt(leftPosition) == 0x25cc; + boolean rightBase = right == LineBreakData.AK || right == LineBreakData.AS || lineCodePointAt(rightPosition) == 0x25cc; + if (left == LineBreakData.AP && rightBase) return true; + if (leftBase && (right == LineBreakData.VF || right == LineBreakData.VI)) return true; + if (left == LineBreakData.VI && (right == LineBreakData.AK || lineCodePointAt(rightPosition) == 0x25cc)) { + int previous = previousLinePosition(leftPosition); + if (previous >= str) { + short value = lineClassAt(previous); + if (value == LineBreakData.AK || value == LineBreakData.AS || lineCodePointAt(previous) == 0x25cc) return true; + } + } + if (leftBase && rightBase) { + int next = nextLinePositionAfter(rightPosition); + return lineClassAt(next) == LineBreakData.VF; + } + return false; + } + + private int lineCodePointAt(int position) { + return position >= str && position < end ? enc.mbcToCode(bytes, position, end) : -1; + } + + private int precedingLineRun(int position, short value) { + int count = 0; + int cursor = position; + while (cursor >= str && lineClassAt(cursor) == value) { + count++; + cursor = previousLinePosition(cursor); + } + return count; + } + private boolean isSentenceBoundary() { if (s <= str || s >= end) return true; // SB1, SB2 diff --git a/third_party/joni/src/org/joni/Lexer.java b/third_party/joni/src/org/joni/Lexer.java index 109acc1fbf..8d26e879ba 100644 --- a/third_party/joni/src/org/joni/Lexer.java +++ b/third_party/joni/src/org/joni/Lexer.java @@ -1454,6 +1454,12 @@ private boolean fetchTokenForPerlBoundary(boolean negated) { : AnchorType.WORD_BREAK_BOUNDARY); return true; } + if (name.toString().equals("lb")) { + fetchTokenFor_anchor(negated + ? AnchorType.NOT_LINE_BOUNDARY + : AnchorType.LINE_BOUNDARY); + return true; + } restore(); return false; } diff --git a/third_party/joni/src/org/joni/LineBreakData.java b/third_party/joni/src/org/joni/LineBreakData.java new file mode 100644 index 0000000000..36b0a244f9 --- /dev/null +++ b/third_party/joni/src/org/joni/LineBreakData.java @@ -0,0 +1,568 @@ +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni; + +// Generated by dev/tools/generate_joni_line_break_data.pl. Do not edit manually. +final class LineBreakData { + static final String UNICODE_VERSION = "17.0.0"; + static final short AI = 0; + static final short AK = 1; + static final short AL = 2; + static final short AP = 3; + static final short AS = 4; + static final short B2 = 5; + static final short BA = 6; + static final short BB = 7; + static final short BK = 8; + static final short CB = 9; + static final short CJ = 10; + static final short CL = 11; + static final short CM = 12; + static final short CP = 13; + static final short CR = 14; + static final short EB = 15; + static final short EM = 16; + static final short EX = 17; + static final short GL = 18; + static final short H2 = 19; + static final short H3 = 20; + static final short HH = 21; + static final short HL = 22; + static final short HY = 23; + static final short ID = 24; + static final short IN = 25; + static final short IS = 26; + static final short JL = 27; + static final short JT = 28; + static final short JV = 29; + static final short LF = 30; + static final short NL = 31; + static final short NS = 32; + static final short NU = 33; + static final short OP = 34; + static final short PO = 35; + static final short PR = 36; + static final short QU = 37; + static final short RI = 38; + static final short SA = 39; + static final short SG = 40; + static final short SP = 41; + static final short SY = 42; + static final short VF = 43; + static final short VI = 44; + static final short WJ = 45; + static final short XX = 46; + static final short ZW = 47; + static final short ZWJ = 48; + static final short CLASS_MASK = 0x3f; + static final short INITIAL_PUNCTUATION = 1 << 6; + static final short FINAL_PUNCTUATION = 1 << 7; + static final short EAST_ASIAN = 1 << 8; + static final short UNASSIGNED_EXTENDED_PICTOGRAPHIC = 1 << 9; + + private static final int[] STARTS = { + 0x0, 0x9, 0xA, 0xB, 0xD, 0xE, 0x20, 0x21, 0x22, 0x23, + 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2A, 0x2B, 0x2C, 0x2D, + 0x2E, 0x2F, 0x30, 0x3A, 0x3C, 0x3F, 0x40, 0x5B, 0x5C, 0x5D, + 0x5E, 0x7B, 0x7C, 0x7D, 0x7E, 0x7F, 0x85, 0x86, 0xA0, 0xA1, + 0xA2, 0xA3, 0xA6, 0xAB, 0xAC, 0xAD, 0xAE, 0xB0, 0xB1, 0xB2, + 0xB4, 0xB5, 0xBB, 0xBC, 0xBF, 0xC0, 0x2C8, 0x2C9, 0x2CC, 0x2CD, + 0x2DF, 0x2E0, 0x300, 0x35C, 0x363, 0x370, 0x37E, 0x37F, 0x483, 0x48A, + 0x589, 0x58A, 0x58B, 0x58F, 0x590, 0x591, 0x5BE, 0x5BF, 0x5C0, 0x5C1, + 0x5C3, 0x5C4, 0x5C6, 0x5C7, 0x5C8, 0x5D0, 0x5EB, 0x5EF, 0x5F3, 0x600, + 0x606, 0x609, 0x60C, 0x60E, 0x610, 0x61B, 0x61C, 0x61D, 0x620, 0x64B, + 0x660, 0x66A, 0x66B, 0x66D, 0x670, 0x671, 0x6D4, 0x6D5, 0x6D6, 0x6DD, + 0x6DE, 0x6DF, 0x6E5, 0x6E7, 0x6E9, 0x6EA, 0x6EE, 0x6F0, 0x6FA, 0x711, + 0x712, 0x730, 0x74B, 0x7A6, 0x7B1, 0x7C0, 0x7CA, 0x7EB, 0x7F4, 0x7F8, + 0x7F9, 0x7FA, 0x7FD, 0x7FE, 0x800, 0x816, 0x81A, 0x81B, 0x824, 0x825, + 0x828, 0x829, 0x82E, 0x859, 0x85C, 0x890, 0x892, 0x897, 0x8A0, 0x8CA, + 0x8E2, 0x8E3, 0x904, 0x93A, 0x93D, 0x93E, 0x950, 0x951, 0x958, 0x962, + 0x964, 0x966, 0x970, 0x981, 0x984, 0x9BC, 0x9BD, 0x9BE, 0x9C5, 0x9C7, + 0x9C9, 0x9CB, 0x9CE, 0x9D7, 0x9D8, 0x9E2, 0x9E4, 0x9E6, 0x9F0, 0x9F2, + 0x9F4, 0x9F9, 0x9FA, 0x9FB, 0x9FC, 0x9FE, 0x9FF, 0xA01, 0xA04, 0xA3C, + 0xA3D, 0xA3E, 0xA43, 0xA47, 0xA49, 0xA4B, 0xA4E, 0xA51, 0xA52, 0xA66, + 0xA70, 0xA72, 0xA75, 0xA76, 0xA81, 0xA84, 0xABC, 0xABD, 0xABE, 0xAC6, + 0xAC7, 0xACA, 0xACB, 0xACE, 0xAE2, 0xAE4, 0xAE6, 0xAF0, 0xAF1, 0xAF2, + 0xAFA, 0xB00, 0xB01, 0xB04, 0xB3C, 0xB3D, 0xB3E, 0xB45, 0xB47, 0xB49, + 0xB4B, 0xB4E, 0xB55, 0xB58, 0xB62, 0xB64, 0xB66, 0xB70, 0xB82, 0xB83, + 0xBBE, 0xBC3, 0xBC6, 0xBC9, 0xBCA, 0xBCE, 0xBD7, 0xBD8, 0xBE6, 0xBF0, + 0xBF9, 0xBFA, 0xC00, 0xC05, 0xC3C, 0xC3D, 0xC3E, 0xC45, 0xC46, 0xC49, + 0xC4A, 0xC4E, 0xC55, 0xC57, 0xC62, 0xC64, 0xC66, 0xC70, 0xC77, 0xC78, + 0xC81, 0xC84, 0xC85, 0xCBC, 0xCBD, 0xCBE, 0xCC5, 0xCC6, 0xCC9, 0xCCA, + 0xCCE, 0xCD5, 0xCD7, 0xCE2, 0xCE4, 0xCE6, 0xCF0, 0xCF3, 0xCF4, 0xD00, + 0xD04, 0xD3B, 0xD3D, 0xD3E, 0xD45, 0xD46, 0xD49, 0xD4A, 0xD4E, 0xD57, + 0xD58, 0xD62, 0xD64, 0xD66, 0xD70, 0xD79, 0xD7A, 0xD81, 0xD84, 0xDCA, + 0xDCB, 0xDCF, 0xDD5, 0xDD6, 0xDD7, 0xDD8, 0xDE0, 0xDE6, 0xDF0, 0xDF2, + 0xDF4, 0xE31, 0xE32, 0xE34, 0xE3B, 0xE3F, 0xE40, 0xE47, 0xE4F, 0xE50, + 0xE5A, 0xE5C, 0xEB1, 0xEB2, 0xEB4, 0xEBD, 0xEC8, 0xECF, 0xED0, 0xEDA, + 0xF01, 0xF05, 0xF06, 0xF08, 0xF09, 0xF0B, 0xF0C, 0xF0D, 0xF12, 0xF13, + 0xF14, 0xF15, 0xF18, 0xF1A, 0xF20, 0xF2A, 0xF34, 0xF35, 0xF36, 0xF37, + 0xF38, 0xF39, 0xF3A, 0xF3B, 0xF3C, 0xF3D, 0xF3E, 0xF40, 0xF71, 0xF7F, + 0xF80, 0xF85, 0xF86, 0xF88, 0xF8D, 0xF98, 0xF99, 0xFBD, 0xFBE, 0xFC0, + 0xFC6, 0xFC7, 0xFD0, 0xFD2, 0xFD3, 0xFD4, 0xFD9, 0xFDB, 0x102B, 0x103F, + 0x1040, 0x104A, 0x104C, 0x1056, 0x105A, 0x105E, 0x1061, 0x1062, 0x1065, 0x1067, + 0x106E, 0x1071, 0x1075, 0x1082, 0x108E, 0x108F, 0x1090, 0x109A, 0x109E, 0x1100, + 0x1160, 0x11A8, 0x1200, 0x135D, 0x1360, 0x1361, 0x1362, 0x1400, 0x1401, 0x1680, + 0x1681, 0x169B, 0x169C, 0x169D, 0x16EB, 0x16EE, 0x1712, 0x1716, 0x1732, 0x1735, + 0x1737, 0x1752, 0x1754, 0x1772, 0x1774, 0x17B4, 0x17D4, 0x17D6, 0x17D7, 0x17D8, + 0x17D9, 0x17DA, 0x17DB, 0x17DC, 0x17DD, 0x17DE, 0x17E0, 0x17EA, 0x1802, 0x1804, + 0x1806, 0x1807, 0x1808, 0x180A, 0x180B, 0x180E, 0x180F, 0x1810, 0x181A, 0x1885, + 0x1887, 0x18A9, 0x18AA, 0x1920, 0x192C, 0x1930, 0x193C, 0x1944, 0x1946, 0x1950, + 0x19D0, 0x19DB, 0x1A17, 0x1A1C, 0x1A55, 0x1A5F, 0x1A60, 0x1A7D, 0x1A7F, 0x1A80, + 0x1A8A, 0x1A90, 0x1A9A, 0x1AB0, 0x1ADE, 0x1AE0, 0x1AEB, 0x1AEC, 0x1B00, 0x1B05, + 0x1B34, 0x1B44, 0x1B45, 0x1B4D, 0x1B4E, 0x1B50, 0x1B5A, 0x1B5C, 0x1B5D, 0x1B61, + 0x1B6B, 0x1B74, 0x1B7D, 0x1B80, 0x1B83, 0x1BA1, 0x1BAE, 0x1BB0, 0x1BBA, 0x1BC0, + 0x1BE6, 0x1BF2, 0x1BF4, 0x1C24, 0x1C38, 0x1C3B, 0x1C40, 0x1C4A, 0x1C50, 0x1C5A, + 0x1C7E, 0x1C80, 0x1CD0, 0x1CD3, 0x1CD4, 0x1CE9, 0x1CED, 0x1CEE, 0x1CF4, 0x1CF5, + 0x1CF7, 0x1CFA, 0x1DC0, 0x1DCD, 0x1DCE, 0x1DFC, 0x1DFD, 0x1E00, 0x1FFD, 0x1FFE, + 0x2000, 0x2007, 0x2008, 0x200B, 0x200C, 0x200D, 0x200E, 0x2010, 0x2011, 0x2012, + 0x2014, 0x2015, 0x2018, 0x2019, 0x201A, 0x201B, 0x201D, 0x201E, 0x201F, 0x2020, + 0x2024, 0x2027, 0x2028, 0x202A, 0x202F, 0x2030, 0x2038, 0x2039, 0x203A, 0x203B, + 0x203C, 0x203E, 0x2044, 0x2045, 0x2046, 0x2047, 0x204A, 0x2056, 0x2057, 0x2058, + 0x205C, 0x205D, 0x2060, 0x2061, 0x2066, 0x2070, 0x207D, 0x207E, 0x207F, 0x208D, + 0x208E, 0x208F, 0x20A0, 0x20A7, 0x20A8, 0x20A9, 0x20AA, 0x20B6, 0x20B7, 0x20BB, + 0x20BC, 0x20BE, 0x20BF, 0x20C0, 0x20C1, 0x20D0, 0x20F1, 0x2103, 0x2104, 0x2109, + 0x210A, 0x2116, 0x2117, 0x2212, 0x2214, 0x22EF, 0x22F0, 0x2308, 0x2309, 0x230A, + 0x230B, 0x230C, 0x231A, 0x231C, 0x2329, 0x232A, 0x232B, 0x23E9, 0x23ED, 0x23F0, + 0x23F1, 0x23F3, 0x23F4, 0x25FD, 0x25FF, 0x2600, 0x2604, 0x2614, 0x2616, 0x2618, + 0x2619, 0x261A, 0x261D, 0x261E, 0x2620, 0x2630, 0x2638, 0x2639, 0x263C, 0x2648, + 0x2654, 0x2668, 0x2669, 0x267F, 0x2680, 0x268A, 0x2690, 0x2693, 0x2694, 0x26A1, + 0x26A2, 0x26AA, 0x26AC, 0x26BD, 0x26BF, 0x26C4, 0x26C6, 0x26C9, 0x26CD, 0x26CE, + 0x26CF, 0x26D2, 0x26D3, 0x26D4, 0x26D5, 0x26D8, 0x26DA, 0x26DC, 0x26DD, 0x26DF, + 0x26E2, 0x26EA, 0x26EB, 0x26F1, 0x26F2, 0x26F4, 0x26F5, 0x26F6, 0x26F7, 0x26F9, + 0x26FA, 0x26FB, 0x26FD, 0x26FE, 0x2705, 0x2706, 0x2708, 0x270A, 0x270C, 0x270E, + 0x2728, 0x2729, 0x274C, 0x274D, 0x274E, 0x274F, 0x2753, 0x2756, 0x2757, 0x2758, + 0x275B, 0x2761, 0x2762, 0x2764, 0x2765, 0x2768, 0x2769, 0x276A, 0x276B, 0x276C, + 0x276D, 0x276E, 0x276F, 0x2770, 0x2771, 0x2772, 0x2773, 0x2774, 0x2775, 0x2776, + 0x2795, 0x2798, 0x27B0, 0x27B1, 0x27BF, 0x27C0, 0x27C5, 0x27C6, 0x27C7, 0x27E6, + 0x27E7, 0x27E8, 0x27E9, 0x27EA, 0x27EB, 0x27EC, 0x27ED, 0x27EE, 0x27EF, 0x27F0, + 0x2800, 0x2801, 0x2983, 0x2984, 0x2985, 0x2986, 0x2987, 0x2988, 0x2989, 0x298A, + 0x298B, 0x298C, 0x298D, 0x298E, 0x298F, 0x2990, 0x2991, 0x2992, 0x2993, 0x2994, + 0x2995, 0x2996, 0x2997, 0x2998, 0x2999, 0x29D8, 0x29D9, 0x29DA, 0x29DB, 0x29DC, + 0x29FC, 0x29FD, 0x29FE, 0x2B1B, 0x2B1D, 0x2B50, 0x2B51, 0x2B55, 0x2B56, 0x2CEF, + 0x2CF2, 0x2CF9, 0x2CFA, 0x2CFD, 0x2CFE, 0x2CFF, 0x2D00, 0x2D70, 0x2D71, 0x2D7F, + 0x2D80, 0x2DE0, 0x2E00, 0x2E02, 0x2E03, 0x2E04, 0x2E05, 0x2E06, 0x2E09, 0x2E0A, + 0x2E0B, 0x2E0C, 0x2E0D, 0x2E0E, 0x2E16, 0x2E17, 0x2E18, 0x2E19, 0x2E1A, 0x2E1C, + 0x2E1D, 0x2E1E, 0x2E20, 0x2E21, 0x2E22, 0x2E23, 0x2E24, 0x2E25, 0x2E26, 0x2E27, + 0x2E28, 0x2E29, 0x2E2A, 0x2E2E, 0x2E2F, 0x2E30, 0x2E32, 0x2E33, 0x2E35, 0x2E3A, + 0x2E3C, 0x2E3F, 0x2E40, 0x2E41, 0x2E42, 0x2E43, 0x2E4B, 0x2E4C, 0x2E4D, 0x2E4E, + 0x2E50, 0x2E53, 0x2E55, 0x2E56, 0x2E57, 0x2E58, 0x2E59, 0x2E5A, 0x2E5B, 0x2E5C, + 0x2E5D, 0x2E5E, 0x2E80, 0x2E9A, 0x2E9B, 0x2EF4, 0x2F00, 0x2FD6, 0x2FF0, 0x3000, + 0x3001, 0x3003, 0x3005, 0x3006, 0x3008, 0x3009, 0x300A, 0x300B, 0x300C, 0x300D, + 0x300E, 0x300F, 0x3010, 0x3011, 0x3012, 0x3014, 0x3015, 0x3016, 0x3017, 0x3018, + 0x3019, 0x301A, 0x301B, 0x301C, 0x301D, 0x301E, 0x3020, 0x302A, 0x3030, 0x3035, + 0x3036, 0x303B, 0x303D, 0x303F, 0x3040, 0x3041, 0x3042, 0x3043, 0x3044, 0x3045, + 0x3046, 0x3047, 0x3048, 0x3049, 0x304A, 0x3063, 0x3064, 0x3083, 0x3084, 0x3085, + 0x3086, 0x3087, 0x3088, 0x308E, 0x308F, 0x3095, 0x3097, 0x3099, 0x309B, 0x309F, + 0x30A0, 0x30A2, 0x30A3, 0x30A4, 0x30A5, 0x30A6, 0x30A7, 0x30A8, 0x30A9, 0x30AA, + 0x30C3, 0x30C4, 0x30E3, 0x30E4, 0x30E5, 0x30E6, 0x30E7, 0x30E8, 0x30EE, 0x30EF, + 0x30F5, 0x30F7, 0x30FB, 0x30FF, 0x3100, 0x3105, 0x3130, 0x3131, 0x318F, 0x3190, + 0x31E6, 0x31EF, 0x31F0, 0x3200, 0x321F, 0x3220, 0x3248, 0x3250, 0x4DC0, 0x4E00, + 0xA015, 0xA016, 0xA48D, 0xA490, 0xA4C7, 0xA4FE, 0xA500, 0xA60D, 0xA60E, 0xA60F, + 0xA610, 0xA620, 0xA62A, 0xA66F, 0xA673, 0xA674, 0xA67E, 0xA69E, 0xA6A0, 0xA6F0, + 0xA6F2, 0xA6F3, 0xA6F8, 0xA802, 0xA803, 0xA806, 0xA807, 0xA80B, 0xA80C, 0xA823, + 0xA828, 0xA82C, 0xA82D, 0xA838, 0xA839, 0xA874, 0xA876, 0xA878, 0xA880, 0xA882, + 0xA8B4, 0xA8C6, 0xA8CE, 0xA8D0, 0xA8DA, 0xA8E0, 0xA8F2, 0xA8FC, 0xA8FD, 0xA8FF, + 0xA900, 0xA90A, 0xA926, 0xA92E, 0xA930, 0xA947, 0xA954, 0xA960, 0xA97D, 0xA980, + 0xA984, 0xA9B3, 0xA9C0, 0xA9C1, 0xA9C7, 0xA9CA, 0xA9CE, 0xA9CF, 0xA9D0, 0xA9DA, + 0xA9DE, 0xA9E0, 0xA9E5, 0xA9E6, 0xA9F0, 0xA9FA, 0xAA00, 0xAA29, 0xAA37, 0xAA40, + 0xAA43, 0xAA44, 0xAA4C, 0xAA4E, 0xAA50, 0xAA5A, 0xAA5C, 0xAA5D, 0xAA60, 0xAA7B, + 0xAA7E, 0xAAB0, 0xAAB1, 0xAAB2, 0xAAB5, 0xAAB7, 0xAAB9, 0xAABE, 0xAAC0, 0xAAC1, + 0xAAC2, 0xAAEB, 0xAAF0, 0xAAF2, 0xAAF5, 0xAAF7, 0xABE3, 0xABEB, 0xABEC, 0xABEE, + 0xABF0, 0xABFA, 0xAC00, 0xAC01, 0xAC1C, 0xAC1D, 0xAC38, 0xAC39, 0xAC54, 0xAC55, + 0xAC70, 0xAC71, 0xAC8C, 0xAC8D, 0xACA8, 0xACA9, 0xACC4, 0xACC5, 0xACE0, 0xACE1, + 0xACFC, 0xACFD, 0xAD18, 0xAD19, 0xAD34, 0xAD35, 0xAD50, 0xAD51, 0xAD6C, 0xAD6D, + 0xAD88, 0xAD89, 0xADA4, 0xADA5, 0xADC0, 0xADC1, 0xADDC, 0xADDD, 0xADF8, 0xADF9, + 0xAE14, 0xAE15, 0xAE30, 0xAE31, 0xAE4C, 0xAE4D, 0xAE68, 0xAE69, 0xAE84, 0xAE85, + 0xAEA0, 0xAEA1, 0xAEBC, 0xAEBD, 0xAED8, 0xAED9, 0xAEF4, 0xAEF5, 0xAF10, 0xAF11, + 0xAF2C, 0xAF2D, 0xAF48, 0xAF49, 0xAF64, 0xAF65, 0xAF80, 0xAF81, 0xAF9C, 0xAF9D, + 0xAFB8, 0xAFB9, 0xAFD4, 0xAFD5, 0xAFF0, 0xAFF1, 0xB00C, 0xB00D, 0xB028, 0xB029, + 0xB044, 0xB045, 0xB060, 0xB061, 0xB07C, 0xB07D, 0xB098, 0xB099, 0xB0B4, 0xB0B5, + 0xB0D0, 0xB0D1, 0xB0EC, 0xB0ED, 0xB108, 0xB109, 0xB124, 0xB125, 0xB140, 0xB141, + 0xB15C, 0xB15D, 0xB178, 0xB179, 0xB194, 0xB195, 0xB1B0, 0xB1B1, 0xB1CC, 0xB1CD, + 0xB1E8, 0xB1E9, 0xB204, 0xB205, 0xB220, 0xB221, 0xB23C, 0xB23D, 0xB258, 0xB259, + 0xB274, 0xB275, 0xB290, 0xB291, 0xB2AC, 0xB2AD, 0xB2C8, 0xB2C9, 0xB2E4, 0xB2E5, + 0xB300, 0xB301, 0xB31C, 0xB31D, 0xB338, 0xB339, 0xB354, 0xB355, 0xB370, 0xB371, + 0xB38C, 0xB38D, 0xB3A8, 0xB3A9, 0xB3C4, 0xB3C5, 0xB3E0, 0xB3E1, 0xB3FC, 0xB3FD, + 0xB418, 0xB419, 0xB434, 0xB435, 0xB450, 0xB451, 0xB46C, 0xB46D, 0xB488, 0xB489, + 0xB4A4, 0xB4A5, 0xB4C0, 0xB4C1, 0xB4DC, 0xB4DD, 0xB4F8, 0xB4F9, 0xB514, 0xB515, + 0xB530, 0xB531, 0xB54C, 0xB54D, 0xB568, 0xB569, 0xB584, 0xB585, 0xB5A0, 0xB5A1, + 0xB5BC, 0xB5BD, 0xB5D8, 0xB5D9, 0xB5F4, 0xB5F5, 0xB610, 0xB611, 0xB62C, 0xB62D, + 0xB648, 0xB649, 0xB664, 0xB665, 0xB680, 0xB681, 0xB69C, 0xB69D, 0xB6B8, 0xB6B9, + 0xB6D4, 0xB6D5, 0xB6F0, 0xB6F1, 0xB70C, 0xB70D, 0xB728, 0xB729, 0xB744, 0xB745, + 0xB760, 0xB761, 0xB77C, 0xB77D, 0xB798, 0xB799, 0xB7B4, 0xB7B5, 0xB7D0, 0xB7D1, + 0xB7EC, 0xB7ED, 0xB808, 0xB809, 0xB824, 0xB825, 0xB840, 0xB841, 0xB85C, 0xB85D, + 0xB878, 0xB879, 0xB894, 0xB895, 0xB8B0, 0xB8B1, 0xB8CC, 0xB8CD, 0xB8E8, 0xB8E9, + 0xB904, 0xB905, 0xB920, 0xB921, 0xB93C, 0xB93D, 0xB958, 0xB959, 0xB974, 0xB975, + 0xB990, 0xB991, 0xB9AC, 0xB9AD, 0xB9C8, 0xB9C9, 0xB9E4, 0xB9E5, 0xBA00, 0xBA01, + 0xBA1C, 0xBA1D, 0xBA38, 0xBA39, 0xBA54, 0xBA55, 0xBA70, 0xBA71, 0xBA8C, 0xBA8D, + 0xBAA8, 0xBAA9, 0xBAC4, 0xBAC5, 0xBAE0, 0xBAE1, 0xBAFC, 0xBAFD, 0xBB18, 0xBB19, + 0xBB34, 0xBB35, 0xBB50, 0xBB51, 0xBB6C, 0xBB6D, 0xBB88, 0xBB89, 0xBBA4, 0xBBA5, + 0xBBC0, 0xBBC1, 0xBBDC, 0xBBDD, 0xBBF8, 0xBBF9, 0xBC14, 0xBC15, 0xBC30, 0xBC31, + 0xBC4C, 0xBC4D, 0xBC68, 0xBC69, 0xBC84, 0xBC85, 0xBCA0, 0xBCA1, 0xBCBC, 0xBCBD, + 0xBCD8, 0xBCD9, 0xBCF4, 0xBCF5, 0xBD10, 0xBD11, 0xBD2C, 0xBD2D, 0xBD48, 0xBD49, + 0xBD64, 0xBD65, 0xBD80, 0xBD81, 0xBD9C, 0xBD9D, 0xBDB8, 0xBDB9, 0xBDD4, 0xBDD5, + 0xBDF0, 0xBDF1, 0xBE0C, 0xBE0D, 0xBE28, 0xBE29, 0xBE44, 0xBE45, 0xBE60, 0xBE61, + 0xBE7C, 0xBE7D, 0xBE98, 0xBE99, 0xBEB4, 0xBEB5, 0xBED0, 0xBED1, 0xBEEC, 0xBEED, + 0xBF08, 0xBF09, 0xBF24, 0xBF25, 0xBF40, 0xBF41, 0xBF5C, 0xBF5D, 0xBF78, 0xBF79, + 0xBF94, 0xBF95, 0xBFB0, 0xBFB1, 0xBFCC, 0xBFCD, 0xBFE8, 0xBFE9, 0xC004, 0xC005, + 0xC020, 0xC021, 0xC03C, 0xC03D, 0xC058, 0xC059, 0xC074, 0xC075, 0xC090, 0xC091, + 0xC0AC, 0xC0AD, 0xC0C8, 0xC0C9, 0xC0E4, 0xC0E5, 0xC100, 0xC101, 0xC11C, 0xC11D, + 0xC138, 0xC139, 0xC154, 0xC155, 0xC170, 0xC171, 0xC18C, 0xC18D, 0xC1A8, 0xC1A9, + 0xC1C4, 0xC1C5, 0xC1E0, 0xC1E1, 0xC1FC, 0xC1FD, 0xC218, 0xC219, 0xC234, 0xC235, + 0xC250, 0xC251, 0xC26C, 0xC26D, 0xC288, 0xC289, 0xC2A4, 0xC2A5, 0xC2C0, 0xC2C1, + 0xC2DC, 0xC2DD, 0xC2F8, 0xC2F9, 0xC314, 0xC315, 0xC330, 0xC331, 0xC34C, 0xC34D, + 0xC368, 0xC369, 0xC384, 0xC385, 0xC3A0, 0xC3A1, 0xC3BC, 0xC3BD, 0xC3D8, 0xC3D9, + 0xC3F4, 0xC3F5, 0xC410, 0xC411, 0xC42C, 0xC42D, 0xC448, 0xC449, 0xC464, 0xC465, + 0xC480, 0xC481, 0xC49C, 0xC49D, 0xC4B8, 0xC4B9, 0xC4D4, 0xC4D5, 0xC4F0, 0xC4F1, + 0xC50C, 0xC50D, 0xC528, 0xC529, 0xC544, 0xC545, 0xC560, 0xC561, 0xC57C, 0xC57D, + 0xC598, 0xC599, 0xC5B4, 0xC5B5, 0xC5D0, 0xC5D1, 0xC5EC, 0xC5ED, 0xC608, 0xC609, + 0xC624, 0xC625, 0xC640, 0xC641, 0xC65C, 0xC65D, 0xC678, 0xC679, 0xC694, 0xC695, + 0xC6B0, 0xC6B1, 0xC6CC, 0xC6CD, 0xC6E8, 0xC6E9, 0xC704, 0xC705, 0xC720, 0xC721, + 0xC73C, 0xC73D, 0xC758, 0xC759, 0xC774, 0xC775, 0xC790, 0xC791, 0xC7AC, 0xC7AD, + 0xC7C8, 0xC7C9, 0xC7E4, 0xC7E5, 0xC800, 0xC801, 0xC81C, 0xC81D, 0xC838, 0xC839, + 0xC854, 0xC855, 0xC870, 0xC871, 0xC88C, 0xC88D, 0xC8A8, 0xC8A9, 0xC8C4, 0xC8C5, + 0xC8E0, 0xC8E1, 0xC8FC, 0xC8FD, 0xC918, 0xC919, 0xC934, 0xC935, 0xC950, 0xC951, + 0xC96C, 0xC96D, 0xC988, 0xC989, 0xC9A4, 0xC9A5, 0xC9C0, 0xC9C1, 0xC9DC, 0xC9DD, + 0xC9F8, 0xC9F9, 0xCA14, 0xCA15, 0xCA30, 0xCA31, 0xCA4C, 0xCA4D, 0xCA68, 0xCA69, + 0xCA84, 0xCA85, 0xCAA0, 0xCAA1, 0xCABC, 0xCABD, 0xCAD8, 0xCAD9, 0xCAF4, 0xCAF5, + 0xCB10, 0xCB11, 0xCB2C, 0xCB2D, 0xCB48, 0xCB49, 0xCB64, 0xCB65, 0xCB80, 0xCB81, + 0xCB9C, 0xCB9D, 0xCBB8, 0xCBB9, 0xCBD4, 0xCBD5, 0xCBF0, 0xCBF1, 0xCC0C, 0xCC0D, + 0xCC28, 0xCC29, 0xCC44, 0xCC45, 0xCC60, 0xCC61, 0xCC7C, 0xCC7D, 0xCC98, 0xCC99, + 0xCCB4, 0xCCB5, 0xCCD0, 0xCCD1, 0xCCEC, 0xCCED, 0xCD08, 0xCD09, 0xCD24, 0xCD25, + 0xCD40, 0xCD41, 0xCD5C, 0xCD5D, 0xCD78, 0xCD79, 0xCD94, 0xCD95, 0xCDB0, 0xCDB1, + 0xCDCC, 0xCDCD, 0xCDE8, 0xCDE9, 0xCE04, 0xCE05, 0xCE20, 0xCE21, 0xCE3C, 0xCE3D, + 0xCE58, 0xCE59, 0xCE74, 0xCE75, 0xCE90, 0xCE91, 0xCEAC, 0xCEAD, 0xCEC8, 0xCEC9, + 0xCEE4, 0xCEE5, 0xCF00, 0xCF01, 0xCF1C, 0xCF1D, 0xCF38, 0xCF39, 0xCF54, 0xCF55, + 0xCF70, 0xCF71, 0xCF8C, 0xCF8D, 0xCFA8, 0xCFA9, 0xCFC4, 0xCFC5, 0xCFE0, 0xCFE1, + 0xCFFC, 0xCFFD, 0xD018, 0xD019, 0xD034, 0xD035, 0xD050, 0xD051, 0xD06C, 0xD06D, + 0xD088, 0xD089, 0xD0A4, 0xD0A5, 0xD0C0, 0xD0C1, 0xD0DC, 0xD0DD, 0xD0F8, 0xD0F9, + 0xD114, 0xD115, 0xD130, 0xD131, 0xD14C, 0xD14D, 0xD168, 0xD169, 0xD184, 0xD185, + 0xD1A0, 0xD1A1, 0xD1BC, 0xD1BD, 0xD1D8, 0xD1D9, 0xD1F4, 0xD1F5, 0xD210, 0xD211, + 0xD22C, 0xD22D, 0xD248, 0xD249, 0xD264, 0xD265, 0xD280, 0xD281, 0xD29C, 0xD29D, + 0xD2B8, 0xD2B9, 0xD2D4, 0xD2D5, 0xD2F0, 0xD2F1, 0xD30C, 0xD30D, 0xD328, 0xD329, + 0xD344, 0xD345, 0xD360, 0xD361, 0xD37C, 0xD37D, 0xD398, 0xD399, 0xD3B4, 0xD3B5, + 0xD3D0, 0xD3D1, 0xD3EC, 0xD3ED, 0xD408, 0xD409, 0xD424, 0xD425, 0xD440, 0xD441, + 0xD45C, 0xD45D, 0xD478, 0xD479, 0xD494, 0xD495, 0xD4B0, 0xD4B1, 0xD4CC, 0xD4CD, + 0xD4E8, 0xD4E9, 0xD504, 0xD505, 0xD520, 0xD521, 0xD53C, 0xD53D, 0xD558, 0xD559, + 0xD574, 0xD575, 0xD590, 0xD591, 0xD5AC, 0xD5AD, 0xD5C8, 0xD5C9, 0xD5E4, 0xD5E5, + 0xD600, 0xD601, 0xD61C, 0xD61D, 0xD638, 0xD639, 0xD654, 0xD655, 0xD670, 0xD671, + 0xD68C, 0xD68D, 0xD6A8, 0xD6A9, 0xD6C4, 0xD6C5, 0xD6E0, 0xD6E1, 0xD6FC, 0xD6FD, + 0xD718, 0xD719, 0xD734, 0xD735, 0xD750, 0xD751, 0xD76C, 0xD76D, 0xD788, 0xD789, + 0xD7A4, 0xD7B0, 0xD7C7, 0xD7CB, 0xD7FC, 0xF900, 0xFB00, 0xFB1D, 0xFB1E, 0xFB1F, + 0xFB29, 0xFB2A, 0xFB37, 0xFB38, 0xFB3D, 0xFB3E, 0xFB3F, 0xFB40, 0xFB42, 0xFB43, + 0xFB45, 0xFB46, 0xFB50, 0xFD3E, 0xFD3F, 0xFD40, 0xFDFC, 0xFDFD, 0xFE00, 0xFE10, + 0xFE13, 0xFE15, 0xFE17, 0xFE18, 0xFE19, 0xFE1A, 0xFE20, 0xFE21, 0xFE22, 0xFE23, + 0xFE24, 0xFE25, 0xFE26, 0xFE28, 0xFE29, 0xFE2A, 0xFE2B, 0xFE2C, 0xFE2D, 0xFE2F, + 0xFE30, 0xFE35, 0xFE36, 0xFE37, 0xFE38, 0xFE39, 0xFE3A, 0xFE3B, 0xFE3C, 0xFE3D, + 0xFE3E, 0xFE3F, 0xFE40, 0xFE41, 0xFE42, 0xFE43, 0xFE44, 0xFE45, 0xFE47, 0xFE48, + 0xFE49, 0xFE50, 0xFE51, 0xFE52, 0xFE53, 0xFE54, 0xFE56, 0xFE58, 0xFE59, 0xFE5A, + 0xFE5B, 0xFE5C, 0xFE5D, 0xFE5E, 0xFE5F, 0xFE67, 0xFE68, 0xFE69, 0xFE6A, 0xFE6B, + 0xFE6C, 0xFEFF, 0xFF00, 0xFF01, 0xFF02, 0xFF04, 0xFF05, 0xFF06, 0xFF08, 0xFF09, + 0xFF0A, 0xFF0C, 0xFF0D, 0xFF0E, 0xFF0F, 0xFF1A, 0xFF1C, 0xFF1F, 0xFF20, 0xFF3B, + 0xFF3C, 0xFF3D, 0xFF3E, 0xFF5B, 0xFF5C, 0xFF5D, 0xFF5E, 0xFF5F, 0xFF60, 0xFF62, + 0xFF63, 0xFF65, 0xFF66, 0xFF67, 0xFF71, 0xFF9E, 0xFFA0, 0xFFBF, 0xFFC2, 0xFFC8, + 0xFFCA, 0xFFD0, 0xFFD2, 0xFFD8, 0xFFDA, 0xFFDD, 0xFFE0, 0xFFE1, 0xFFE2, 0xFFE5, + 0xFFE7, 0xFFE8, 0xFFEF, 0xFFF9, 0xFFFC, 0xFFFD, 0x10100, 0x10103, 0x101FD, 0x101FE, + 0x102E0, 0x102E1, 0x10376, 0x1037B, 0x1039F, 0x103A0, 0x103D0, 0x103D1, 0x104A0, 0x104AA, + 0x10857, 0x10858, 0x1091F, 0x10920, 0x10A01, 0x10A04, 0x10A05, 0x10A07, 0x10A0C, 0x10A10, + 0x10A38, 0x10A3B, 0x10A3F, 0x10A40, 0x10A50, 0x10A58, 0x10AE5, 0x10AE7, 0x10AF0, 0x10AF6, + 0x10AF7, 0x10B39, 0x10B40, 0x10D24, 0x10D28, 0x10D30, 0x10D3A, 0x10D40, 0x10D4A, 0x10D69, + 0x10D6E, 0x10D6F, 0x10EAB, 0x10EAD, 0x10EAE, 0x10ED0, 0x10ED1, 0x10EFA, 0x10F00, 0x10F46, + 0x10F51, 0x10F82, 0x10F86, 0x11000, 0x11003, 0x11005, 0x11038, 0x11046, 0x11047, 0x11049, + 0x1104E, 0x11052, 0x11066, 0x11070, 0x11071, 0x11073, 0x11075, 0x11076, 0x1107F, 0x11080, + 0x11083, 0x110B0, 0x110BB, 0x110BD, 0x110BE, 0x110C2, 0x110C3, 0x110CD, 0x110CE, 0x110F0, + 0x110FA, 0x11100, 0x11103, 0x11127, 0x11135, 0x11136, 0x11140, 0x11144, 0x11145, 0x11147, + 0x11173, 0x11174, 0x11175, 0x11176, 0x11180, 0x11183, 0x111B3, 0x111C1, 0x111C5, 0x111C7, + 0x111C8, 0x111C9, 0x111CD, 0x111CE, 0x111D0, 0x111DA, 0x111DB, 0x111DC, 0x111DD, 0x111E0, + 0x1122C, 0x11238, 0x1123A, 0x1123B, 0x1123D, 0x1123E, 0x1123F, 0x11241, 0x11242, 0x112A9, + 0x112AA, 0x112DF, 0x112EB, 0x112F0, 0x112FA, 0x11300, 0x11304, 0x11305, 0x1130D, 0x1130F, + 0x11311, 0x11313, 0x11329, 0x1132A, 0x11331, 0x11332, 0x11334, 0x11335, 0x1133A, 0x1133B, + 0x1133D, 0x1133E, 0x11345, 0x11347, 0x11349, 0x1134B, 0x1134D, 0x1134E, 0x11350, 0x11351, + 0x11357, 0x11358, 0x1135D, 0x1135E, 0x11360, 0x11362, 0x11364, 0x11366, 0x1136D, 0x11370, + 0x11375, 0x11380, 0x1138A, 0x1138B, 0x1138C, 0x1138E, 0x1138F, 0x11390, 0x11392, 0x113B6, + 0x113B7, 0x113B8, 0x113C1, 0x113C2, 0x113C3, 0x113C5, 0x113C6, 0x113C7, 0x113CB, 0x113CC, + 0x113D0, 0x113D1, 0x113D2, 0x113D3, 0x113D6, 0x113D7, 0x113D9, 0x113E1, 0x113E3, 0x11435, + 0x11447, 0x1144B, 0x1144F, 0x11450, 0x1145A, 0x1145C, 0x1145E, 0x1145F, 0x114B0, 0x114C4, + 0x114D0, 0x114DA, 0x115AF, 0x115B6, 0x115B8, 0x115C1, 0x115C2, 0x115C4, 0x115C6, 0x115C9, + 0x115D8, 0x115DC, 0x115DE, 0x11630, 0x11641, 0x11643, 0x11650, 0x1165A, 0x11660, 0x1166D, + 0x116AB, 0x116B8, 0x116C0, 0x116CA, 0x116D0, 0x116E4, 0x1171D, 0x1172C, 0x11730, 0x1173A, + 0x1173C, 0x1173F, 0x1182C, 0x1183B, 0x118E0, 0x118EA, 0x11900, 0x11907, 0x11909, 0x1190A, + 0x1190C, 0x11914, 0x11915, 0x11917, 0x11918, 0x11930, 0x11936, 0x11937, 0x11939, 0x1193B, + 0x1193E, 0x1193F, 0x11940, 0x11941, 0x11942, 0x11944, 0x11947, 0x11950, 0x1195A, 0x119D1, + 0x119D8, 0x119DA, 0x119E1, 0x119E2, 0x119E3, 0x119E4, 0x119E5, 0x11A01, 0x11A0B, 0x11A33, + 0x11A3A, 0x11A3B, 0x11A3F, 0x11A40, 0x11A41, 0x11A45, 0x11A46, 0x11A47, 0x11A48, 0x11A51, + 0x11A5C, 0x11A8A, 0x11A9A, 0x11A9D, 0x11A9E, 0x11AA1, 0x11AA3, 0x11B00, 0x11B0A, 0x11B60, + 0x11B68, 0x11BF0, 0x11BFA, 0x11C2F, 0x11C37, 0x11C38, 0x11C40, 0x11C41, 0x11C46, 0x11C50, + 0x11C5A, 0x11C70, 0x11C71, 0x11C72, 0x11C92, 0x11CA8, 0x11CA9, 0x11CB7, 0x11D31, 0x11D37, + 0x11D3A, 0x11D3B, 0x11D3C, 0x11D3E, 0x11D3F, 0x11D46, 0x11D47, 0x11D48, 0x11D50, 0x11D5A, + 0x11D8A, 0x11D8F, 0x11D90, 0x11D92, 0x11D93, 0x11D98, 0x11DA0, 0x11DAA, 0x11DE0, 0x11DEA, + 0x11EE0, 0x11EF2, 0x11EF3, 0x11EF7, 0x11EF9, 0x11F00, 0x11F02, 0x11F03, 0x11F04, 0x11F11, + 0x11F12, 0x11F34, 0x11F3B, 0x11F3E, 0x11F42, 0x11F43, 0x11F45, 0x11F50, 0x11F5A, 0x11F5B, + 0x11FDD, 0x11FE1, 0x11FFF, 0x12000, 0x12470, 0x12475, 0x13258, 0x1325B, 0x1325E, 0x13282, + 0x13283, 0x13286, 0x13287, 0x13288, 0x13289, 0x1328A, 0x13379, 0x1337A, 0x1337C, 0x1342F, + 0x13430, 0x13437, 0x13438, 0x13439, 0x1343C, 0x1343D, 0x1343E, 0x1343F, 0x13440, 0x13441, + 0x13447, 0x13456, 0x145CE, 0x145CF, 0x145D0, 0x16100, 0x1611E, 0x16130, 0x1613A, 0x16A60, + 0x16A6A, 0x16A6E, 0x16A70, 0x16AC0, 0x16ACA, 0x16AF0, 0x16AF5, 0x16AF6, 0x16B30, 0x16B37, + 0x16B3A, 0x16B44, 0x16B45, 0x16B50, 0x16B5A, 0x16D6E, 0x16D70, 0x16D7A, 0x16E97, 0x16E99, + 0x16F4F, 0x16F50, 0x16F51, 0x16F88, 0x16F8F, 0x16F93, 0x16FE0, 0x16FE4, 0x16FE5, 0x16FF0, + 0x16FF2, 0x16FF4, 0x16FF7, 0x17000, 0x18B00, 0x18CD6, 0x18CFF, 0x18D00, 0x18D1F, 0x18D80, + 0x18DF3, 0x1AFF0, 0x1AFF4, 0x1AFF5, 0x1AFFC, 0x1AFFD, 0x1AFFF, 0x1B000, 0x1B123, 0x1B132, + 0x1B133, 0x1B150, 0x1B153, 0x1B155, 0x1B156, 0x1B164, 0x1B168, 0x1B170, 0x1B2FC, 0x1BC9D, + 0x1BC9F, 0x1BCA0, 0x1BCA4, 0x1CCF0, 0x1CCFA, 0x1CF00, 0x1CF2E, 0x1CF30, 0x1CF47, 0x1D165, + 0x1D16A, 0x1D16D, 0x1D183, 0x1D185, 0x1D18C, 0x1D1AA, 0x1D1AE, 0x1D242, 0x1D245, 0x1D300, + 0x1D357, 0x1D360, 0x1D377, 0x1D7CE, 0x1D800, 0x1DA00, 0x1DA37, 0x1DA3B, 0x1DA6D, 0x1DA75, + 0x1DA76, 0x1DA84, 0x1DA85, 0x1DA87, 0x1DA8B, 0x1DA9B, 0x1DAA0, 0x1DAA1, 0x1DAB0, 0x1E000, + 0x1E007, 0x1E008, 0x1E019, 0x1E01B, 0x1E022, 0x1E023, 0x1E025, 0x1E026, 0x1E02B, 0x1E08F, + 0x1E090, 0x1E130, 0x1E137, 0x1E140, 0x1E14A, 0x1E2AE, 0x1E2AF, 0x1E2EC, 0x1E2F0, 0x1E2FA, + 0x1E2FF, 0x1E300, 0x1E4EC, 0x1E4F0, 0x1E4FA, 0x1E5EE, 0x1E5F0, 0x1E5F1, 0x1E5FB, 0x1E6E3, + 0x1E6E4, 0x1E6E6, 0x1E6E7, 0x1E6EE, 0x1E6F0, 0x1E6F5, 0x1E6F6, 0x1E8D0, 0x1E8D7, 0x1E944, + 0x1E94B, 0x1E950, 0x1E95A, 0x1E95E, 0x1E960, 0x1ECAC, 0x1ECAD, 0x1ECB0, 0x1ECB1, 0x1F000, + 0x1F004, 0x1F005, 0x1F02C, 0x1F030, 0x1F094, 0x1F0A0, 0x1F0AF, 0x1F0B1, 0x1F0C0, 0x1F0C1, + 0x1F0CF, 0x1F0D0, 0x1F0D1, 0x1F0F6, 0x1F100, 0x1F18E, 0x1F18F, 0x1F191, 0x1F19B, 0x1F1AE, + 0x1F1E6, 0x1F200, 0x1F203, 0x1F210, 0x1F23C, 0x1F240, 0x1F249, 0x1F250, 0x1F252, 0x1F260, + 0x1F266, 0x1F300, 0x1F321, 0x1F32D, 0x1F336, 0x1F337, 0x1F37D, 0x1F37E, 0x1F385, 0x1F386, + 0x1F394, 0x1F39C, 0x1F39E, 0x1F3A0, 0x1F3B5, 0x1F3B7, 0x1F3BC, 0x1F3BD, 0x1F3C2, 0x1F3C5, + 0x1F3C7, 0x1F3C8, 0x1F3CA, 0x1F3CB, 0x1F3CD, 0x1F3CF, 0x1F3D4, 0x1F3E0, 0x1F3F1, 0x1F3F4, + 0x1F3F5, 0x1F3F8, 0x1F3FB, 0x1F400, 0x1F43F, 0x1F440, 0x1F441, 0x1F442, 0x1F444, 0x1F446, + 0x1F451, 0x1F466, 0x1F479, 0x1F47C, 0x1F47D, 0x1F481, 0x1F484, 0x1F485, 0x1F488, 0x1F48F, + 0x1F490, 0x1F491, 0x1F492, 0x1F4A0, 0x1F4A1, 0x1F4A2, 0x1F4A3, 0x1F4A4, 0x1F4A5, 0x1F4AA, + 0x1F4AB, 0x1F4AF, 0x1F4B0, 0x1F4B1, 0x1F4B3, 0x1F4FD, 0x1F4FF, 0x1F500, 0x1F507, 0x1F517, + 0x1F525, 0x1F532, 0x1F53E, 0x1F54A, 0x1F54B, 0x1F54F, 0x1F550, 0x1F568, 0x1F574, 0x1F576, + 0x1F57A, 0x1F57B, 0x1F590, 0x1F591, 0x1F595, 0x1F597, 0x1F5A4, 0x1F5A5, 0x1F5D4, 0x1F5DC, + 0x1F5F4, 0x1F5FA, 0x1F5FB, 0x1F645, 0x1F648, 0x1F64B, 0x1F650, 0x1F676, 0x1F679, 0x1F67C, + 0x1F680, 0x1F6A3, 0x1F6A4, 0x1F6B4, 0x1F6B7, 0x1F6C0, 0x1F6C1, 0x1F6C6, 0x1F6CC, 0x1F6CD, + 0x1F6D0, 0x1F6D3, 0x1F6D5, 0x1F6D9, 0x1F6DC, 0x1F6E0, 0x1F6EB, 0x1F6ED, 0x1F6F0, 0x1F6F4, + 0x1F6FD, 0x1F700, 0x1F774, 0x1F777, 0x1F77B, 0x1F780, 0x1F7D5, 0x1F7DA, 0x1F7E0, 0x1F7EC, + 0x1F7F0, 0x1F7F1, 0x1F800, 0x1F80C, 0x1F810, 0x1F848, 0x1F850, 0x1F85A, 0x1F860, 0x1F888, + 0x1F890, 0x1F8AE, 0x1F8B0, 0x1F8BC, 0x1F8C0, 0x1F8C2, 0x1F8D0, 0x1F8D9, 0x1F900, 0x1F90C, + 0x1F90D, 0x1F90F, 0x1F910, 0x1F918, 0x1F920, 0x1F926, 0x1F927, 0x1F930, 0x1F93A, 0x1F93B, + 0x1F93C, 0x1F93F, 0x1F946, 0x1F947, 0x1F977, 0x1F978, 0x1F9B5, 0x1F9B7, 0x1F9B8, 0x1F9BA, + 0x1F9BB, 0x1F9BC, 0x1F9CD, 0x1F9D0, 0x1F9D1, 0x1F9DE, 0x1FA00, 0x1FA58, 0x1FA60, 0x1FA6E, + 0x1FA70, 0x1FA7D, 0x1FA80, 0x1FA8B, 0x1FA8E, 0x1FAC3, 0x1FAC6, 0x1FAC7, 0x1FAC8, 0x1FAC9, + 0x1FACD, 0x1FADD, 0x1FADF, 0x1FAEB, 0x1FAEF, 0x1FAF0, 0x1FAF9, 0x1FB00, 0x1FBF0, 0x1FBFA, + 0x1FC00, 0x1FFFE, 0x20000, 0x2FFFE, 0x30000, 0x3FFFE, 0xE0001, 0xE0002, 0xE0020, 0xE0080, + 0xE0100, 0xE01F0, + }; + + private static final short[] VALUES = { + 12, 6, 30, 8, 14, 12, 41, 17, 37, 2, 36, 35, 2, 37, 34, 13, + 2, 36, 26, 23, 26, 42, 33, 26, 2, 17, 2, 34, 36, 13, 2, 34, + 6, 11, 2, 12, 31, 12, 18, 34, 35, 36, 2, 101, 2, 6, 2, 35, + 36, 2, 7, 2, 165, 2, 34, 2, 7, 2, 7, 2, 7, 2, 12, 18, + 12, 2, 26, 2, 12, 2, 26, 21, 2, 36, 2, 12, 21, 12, 2, 12, + 2, 12, 17, 12, 2, 22, 2, 22, 2, 33, 2, 35, 26, 2, 12, 17, + 12, 17, 2, 12, 33, 35, 33, 2, 12, 2, 17, 2, 12, 33, 2, 12, + 2, 12, 2, 12, 2, 33, 2, 12, 2, 12, 2, 12, 2, 33, 2, 12, + 2, 26, 17, 2, 12, 36, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, + 2, 33, 2, 12, 2, 12, 33, 12, 2, 12, 2, 12, 2, 12, 2, 12, + 6, 33, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, + 2, 33, 2, 35, 2, 35, 2, 36, 2, 12, 2, 12, 2, 12, 2, 12, + 2, 12, 2, 12, 2, 12, 2, 33, 12, 2, 12, 2, 12, 2, 12, 2, + 12, 2, 12, 2, 12, 2, 12, 2, 33, 2, 36, 2, 12, 2, 12, 2, + 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 33, 2, 12, 2, + 12, 2, 12, 2, 12, 2, 12, 2, 33, 2, 36, 2, 12, 2, 12, 2, + 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 33, 2, 7, 2, 12, 7, + 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 33, 2, 12, + 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 33, + 2, 35, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 33, 2, 12, + 2, 12, 2, 12, 2, 36, 2, 12, 2, 33, 6, 2, 12, 2, 12, 2, + 12, 2, 33, 2, 7, 2, 7, 18, 7, 6, 18, 17, 18, 2, 17, 2, + 12, 2, 33, 2, 6, 12, 2, 12, 2, 12, 34, 11, 34, 11, 12, 2, + 12, 6, 12, 6, 12, 2, 12, 2, 12, 2, 6, 2, 12, 2, 7, 6, + 7, 2, 18, 2, 12, 2, 33, 6, 2, 12, 2, 12, 2, 12, 2, 12, + 2, 12, 2, 12, 2, 12, 33, 12, 2, 283, 29, 28, 2, 12, 2, 6, + 2, 21, 2, 6, 2, 34, 11, 2, 6, 2, 12, 2, 12, 6, 2, 12, + 2, 12, 2, 12, 6, 32, 2, 6, 2, 6, 36, 2, 12, 2, 33, 2, + 17, 6, 7, 2, 17, 2, 12, 18, 12, 33, 2, 12, 2, 12, 2, 12, + 2, 12, 2, 17, 33, 2, 33, 2, 12, 2, 12, 2, 12, 2, 12, 33, + 2, 33, 2, 12, 2, 12, 18, 2, 12, 1, 12, 44, 1, 2, 6, 4, + 6, 24, 6, 24, 12, 24, 6, 12, 2, 12, 2, 33, 2, 4, 12, 43, + 2, 12, 2, 6, 33, 2, 33, 2, 6, 2, 12, 2, 12, 2, 12, 2, + 12, 2, 12, 2, 12, 18, 12, 18, 12, 2, 7, 2, 6, 18, 6, 47, + 12, 48, 12, 21, 18, 21, 5, 2, 101, 165, 34, 101, 165, 34, 101, 2, + 25, 6, 8, 12, 18, 35, 2, 101, 165, 2, 32, 2, 26, 34, 11, 32, + 2, 6, 35, 6, 2, 6, 45, 2, 12, 2, 34, 11, 2, 34, 11, 2, + 36, 35, 36, 292, 36, 35, 36, 35, 36, 35, 36, 35, 36, 12, 2, 35, + 2, 35, 2, 36, 2, 36, 2, 25, 2, 34, 11, 34, 11, 2, 280, 2, + 290, 267, 2, 258, 2, 280, 24, 280, 2, 258, 2, 24, 2, 280, 2, 24, + 2, 24, 15, 24, 2, 258, 2, 24, 2, 258, 2, 24, 2, 280, 2, 258, + 2, 258, 2, 258, 2, 258, 2, 280, 24, 280, 24, 2, 24, 258, 24, 2, + 24, 280, 2, 24, 2, 24, 2, 24, 2, 280, 2, 24, 280, 24, 280, 2, + 24, 15, 280, 2, 280, 24, 258, 2, 24, 271, 15, 2, 258, 2, 258, 2, + 258, 2, 258, 2, 258, 2, 37, 2, 17, 24, 2, 34, 11, 34, 11, 34, + 11, 34, 11, 34, 11, 34, 11, 34, 11, 2, 258, 2, 258, 2, 258, 2, + 34, 11, 2, 34, 11, 34, 11, 34, 11, 34, 11, 34, 11, 2, 6, 2, + 34, 11, 34, 11, 34, 11, 34, 11, 34, 11, 34, 11, 34, 11, 34, 11, + 34, 11, 34, 11, 34, 11, 2, 34, 11, 34, 11, 2, 34, 11, 2, 258, + 2, 258, 2, 258, 2, 12, 2, 17, 6, 2, 17, 6, 2, 6, 2, 12, + 2, 12, 37, 101, 165, 101, 165, 37, 101, 165, 37, 101, 165, 6, 2, 21, + 34, 6, 2, 101, 165, 2, 101, 165, 34, 11, 34, 11, 34, 11, 34, 11, + 6, 17, 2, 6, 2, 6, 2, 5, 6, 2, 21, 6, 34, 6, 2, 6, + 2, 6, 2, 17, 34, 13, 34, 13, 34, 13, 34, 13, 21, 2, 280, 2, + 280, 2, 280, 2, 280, 262, 267, 280, 288, 280, 290, 267, 290, 267, 290, 267, + 290, 267, 290, 267, 280, 290, 267, 290, 267, 290, 267, 290, 267, 288, 290, 267, + 280, 268, 280, 268, 280, 288, 280, 24, 2, 288, 280, 288, 280, 288, 280, 288, + 280, 288, 280, 288, 280, 288, 280, 288, 280, 288, 280, 288, 280, 288, 2, 268, + 288, 280, 288, 280, 288, 280, 288, 280, 288, 280, 288, 280, 288, 280, 288, 280, + 288, 280, 288, 280, 288, 280, 288, 280, 288, 280, 2, 280, 2, 280, 2, 280, + 2, 280, 288, 280, 2, 280, 2, 280, 258, 280, 288, 280, 2, 280, 2, 6, + 2, 6, 17, 6, 2, 33, 2, 12, 2, 12, 2, 12, 2, 12, 2, 6, + 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 35, 2, 7, 17, 2, + 12, 2, 12, 2, 6, 33, 2, 12, 2, 7, 2, 12, 33, 2, 12, 6, + 2, 12, 2, 283, 2, 12, 1, 12, 44, 24, 6, 24, 2, 6, 4, 2, + 24, 2, 12, 2, 33, 2, 4, 12, 2, 6, 12, 6, 12, 2, 4, 2, + 24, 6, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, + 6, 2, 12, 2, 12, 6, 12, 2, 33, 2, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, 275, 276, + 275, 276, 275, 276, 275, 276, 275, 276, 2, 29, 2, 28, 2, 280, 2, 22, + 12, 22, 2, 22, 2, 22, 2, 22, 2, 22, 2, 22, 2, 22, 2, 11, + 34, 2, 35, 2, 12, 267, 288, 273, 290, 267, 281, 2, 18, 12, 18, 12, + 18, 12, 18, 12, 18, 12, 18, 12, 18, 12, 280, 290, 267, 290, 267, 290, + 267, 290, 267, 290, 267, 290, 267, 290, 267, 290, 267, 280, 290, 267, 280, 267, + 280, 267, 2, 288, 273, 280, 290, 267, 290, 267, 290, 267, 280, 2, 280, 292, + 291, 280, 2, 45, 2, 273, 280, 292, 291, 280, 290, 267, 280, 267, 280, 267, + 280, 288, 280, 273, 280, 290, 280, 267, 280, 290, 280, 267, 280, 290, 267, 290, + 267, 288, 280, 288, 280, 288, 280, 2, 280, 2, 280, 2, 280, 2, 280, 2, + 291, 292, 280, 292, 2, 258, 2, 12, 9, 2, 6, 2, 12, 2, 12, 2, + 12, 2, 6, 2, 6, 2, 33, 2, 6, 2, 6, 2, 12, 2, 12, 2, + 12, 2, 12, 2, 12, 2, 6, 2, 12, 2, 6, 25, 2, 6, 2, 12, + 2, 33, 2, 33, 2, 12, 21, 2, 12, 21, 2, 6, 2, 12, 2, 12, + 2, 12, 2, 12, 3, 1, 12, 44, 6, 24, 2, 24, 4, 12, 1, 12, + 1, 2, 18, 12, 2, 12, 2, 33, 6, 12, 2, 33, 2, 33, 2, 12, + 2, 12, 2, 33, 6, 2, 12, 2, 12, 2, 7, 2, 12, 2, 12, 2, + 6, 2, 6, 12, 2, 12, 33, 2, 7, 2, 6, 2, 12, 6, 2, 6, + 2, 12, 2, 12, 2, 6, 2, 12, 2, 33, 2, 12, 2, 1, 2, 1, + 2, 1, 2, 1, 2, 1, 2, 1, 2, 12, 6, 12, 2, 12, 2, 12, + 44, 2, 4, 2, 12, 2, 6, 4, 1, 12, 2, 12, 2, 12, 2, 4, + 2, 4, 2, 4, 2, 4, 1, 2, 24, 12, 2, 12, 2, 12, 2, 12, + 2, 12, 44, 3, 12, 24, 2, 24, 2, 12, 2, 12, 2, 6, 2, 33, + 6, 2, 12, 2, 12, 2, 33, 2, 12, 2, 12, 7, 6, 17, 2, 6, + 2, 12, 2, 12, 6, 2, 33, 2, 7, 2, 12, 2, 33, 2, 33, 2, + 12, 2, 33, 2, 6, 2, 12, 2, 33, 2, 1, 2, 1, 2, 1, 2, + 1, 2, 1, 12, 2, 12, 2, 12, 44, 3, 12, 3, 12, 6, 2, 4, + 2, 12, 2, 12, 2, 7, 2, 12, 2, 12, 2, 12, 2, 12, 7, 2, + 6, 7, 2, 12, 2, 12, 2, 12, 6, 2, 7, 6, 2, 7, 2, 12, + 2, 33, 2, 12, 2, 12, 2, 6, 2, 33, 2, 7, 17, 2, 12, 2, + 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 33, 2, 12, 2, + 12, 2, 12, 2, 33, 2, 33, 2, 4, 6, 12, 6, 2, 12, 3, 12, + 1, 2, 1, 12, 2, 12, 44, 6, 24, 4, 12, 2, 35, 2, 6, 2, + 6, 2, 34, 11, 2, 11, 2, 34, 11, 34, 11, 2, 34, 11, 2, 34, + 18, 34, 11, 18, 34, 11, 34, 11, 12, 2, 12, 2, 34, 11, 2, 4, + 12, 4, 2, 33, 2, 6, 2, 33, 2, 12, 6, 2, 12, 6, 2, 6, + 2, 33, 2, 6, 33, 2, 6, 2, 12, 2, 12, 2, 12, 2, 288, 274, + 2, 268, 288, 280, 2, 280, 258, 2, 258, 280, 2, 280, 2, 258, 2, 258, + 2, 258, 2, 280, 2, 288, 2, 288, 2, 288, 2, 288, 2, 280, 2, 12, + 6, 12, 2, 33, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, + 2, 12, 2, 258, 2, 258, 2, 33, 2, 12, 2, 12, 2, 12, 2, 12, + 2, 6, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, + 2, 12, 2, 12, 2, 33, 2, 12, 2, 12, 33, 2, 36, 2, 12, 33, + 2, 12, 2, 33, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, 2, 12, + 2, 33, 2, 34, 2, 35, 2, 35, 2, 24, 280, 24, 536, 24, 536, 24, + 536, 24, 536, 24, 280, 536, 24, 536, 2, 258, 2, 258, 2, 536, 38, 280, + 536, 280, 536, 280, 536, 280, 536, 280, 536, 280, 24, 280, 24, 280, 24, 280, + 271, 280, 24, 2, 24, 280, 258, 280, 258, 280, 271, 280, 271, 280, 271, 15, + 24, 280, 24, 280, 24, 280, 24, 280, 272, 280, 24, 280, 24, 271, 280, 271, + 280, 271, 280, 271, 280, 271, 280, 271, 280, 271, 280, 271, 280, 258, 280, 258, + 280, 258, 280, 271, 280, 258, 280, 258, 280, 24, 280, 258, 280, 258, 280, 258, + 2, 24, 280, 24, 280, 24, 15, 24, 271, 24, 15, 24, 271, 24, 280, 24, + 2, 24, 2, 24, 280, 271, 280, 271, 2, 37, 32, 2, 280, 271, 280, 271, + 280, 271, 280, 24, 271, 24, 280, 24, 280, 536, 280, 24, 280, 536, 24, 280, + 536, 2, 24, 2, 24, 2, 24, 536, 280, 536, 280, 536, 2, 514, 2, 514, + 2, 514, 2, 514, 2, 514, 2, 514, 2, 514, 2, 514, 2, 271, 280, 271, + 280, 271, 280, 271, 280, 271, 280, 24, 271, 280, 24, 280, 271, 280, 271, 280, + 271, 280, 271, 280, 271, 280, 271, 280, 2, 536, 24, 536, 280, 536, 280, 536, + 280, 271, 280, 536, 280, 536, 280, 536, 280, 536, 280, 271, 536, 2, 33, 2, + 536, 2, 280, 2, 280, 2, 12, 2, 12, 2, 12, 2, + }; + + static short propertyOf(int codePoint) { + int low = 0; + int high = STARTS.length - 1; + while (low <= high) { + int middle = (low + high) >>> 1; + if (STARTS[middle] <= codePoint) { + low = middle + 1; + } else { + high = middle - 1; + } + } + return VALUES[Math.max(0, high)]; + } + + static short lineClass(short property) { + return (short)(property & CLASS_MASK); + } + + static boolean has(short property, short flag) { + return (property & flag) != 0; + } + + private LineBreakData() { + } +} diff --git a/third_party/joni/src/org/joni/constants/internal/AnchorType.java b/third_party/joni/src/org/joni/constants/internal/AnchorType.java index f1cb3b99ef..df6138588c 100644 --- a/third_party/joni/src/org/joni/constants/internal/AnchorType.java +++ b/third_party/joni/src/org/joni/constants/internal/AnchorType.java @@ -49,6 +49,8 @@ public interface AnchorType { int NOT_SENTENCE_BOUNDARY = (1<<20); int WORD_BREAK_BOUNDARY = (1<<21); int NOT_WORD_BREAK_BOUNDARY = (1<<22); + int LINE_BOUNDARY = (1<<23); + int NOT_LINE_BOUNDARY = (1<<24); int ALLOWED_IN_LB = ( LOOK_BEHIND | LOOK_BEHIND_NOT | @@ -65,6 +67,8 @@ public interface AnchorType { NOT_SENTENCE_BOUNDARY | WORD_BREAK_BOUNDARY | NOT_WORD_BREAK_BOUNDARY | + LINE_BOUNDARY | + NOT_LINE_BOUNDARY | WORD_BEGIN | WORD_END ); @@ -84,6 +88,8 @@ public interface AnchorType { NOT_SENTENCE_BOUNDARY | WORD_BREAK_BOUNDARY | NOT_WORD_BREAK_BOUNDARY | + LINE_BOUNDARY | + NOT_LINE_BOUNDARY | WORD_BEGIN | WORD_END ); } diff --git a/third_party/joni/src/org/joni/constants/internal/OPCode.java b/third_party/joni/src/org/joni/constants/internal/OPCode.java index fdec60efe3..f9a85e21dd 100644 --- a/third_party/joni/src/org/joni/constants/internal/OPCode.java +++ b/third_party/joni/src/org/joni/constants/internal/OPCode.java @@ -160,6 +160,8 @@ public interface OPCode { int NOT_SENTENCE_BOUNDARY = 116; /* Perl \B{sb} */ int WORD_BREAK_BOUNDARY = 117; /* Perl \b{wb} */ int NOT_WORD_BREAK_BOUNDARY = 118; /* Perl \B{wb} */ + int LINE_BOUNDARY = 119; /* Perl \b{lb} */ + int NOT_LINE_BOUNDARY = 120; /* Perl \B{lb} */ String[] OpCodeNames = Config.DEBUG_COMPILE ? new String[] { "finish", /*OP_FINISH*/ @@ -282,6 +284,8 @@ public interface OPCode { "not-sentence-boundary", /*OP_NOT_SENTENCE_BOUNDARY*/ "word-break-boundary", /*OP_WORD_BREAK_BOUNDARY*/ "not-word-break-boundary", /*OP_NOT_WORD_BREAK_BOUNDARY*/ + "line-boundary", /*OP_LINE_BOUNDARY*/ + "not-line-boundary", /*OP_NOT_LINE_BOUNDARY*/ } : null; int[] OpCodeArgTypes = Config.DEBUG_COMPILE ? new int[] { @@ -405,5 +409,7 @@ public interface OPCode { Arguments.NON, /*OP_NOT_SENTENCE_BOUNDARY*/ Arguments.NON, /*OP_WORD_BREAK_BOUNDARY*/ Arguments.NON, /*OP_NOT_WORD_BREAK_BOUNDARY*/ + Arguments.NON, /*OP_LINE_BOUNDARY*/ + Arguments.NON, /*OP_NOT_LINE_BOUNDARY*/ } : null; } diff --git a/third_party/joni/test/org/joni/test/TestPerlLineBoundary.java b/third_party/joni/test/org/joni/test/TestPerlLineBoundary.java new file mode 100644 index 0000000000..840267c9ca --- /dev/null +++ b/third_party/joni/test/org/joni/test/TestPerlLineBoundary.java @@ -0,0 +1,102 @@ +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni.test; + +import static org.junit.Assert.assertEquals; + +import java.nio.charset.StandardCharsets; + +import org.jcodings.specific.UTF8Encoding; +import org.joni.Option; +import org.joni.Regex; +import org.joni.Syntax; +import org.junit.Test; + +public class TestPerlLineBoundary { + private static int search(String pattern, String input) { + byte[] patternBytes = pattern.getBytes(StandardCharsets.UTF_8); + byte[] inputBytes = input.getBytes(StandardCharsets.UTF_8); + Regex regex = new Regex(patternBytes, 0, patternBytes.length, Option.NONE, + UTF8Encoding.INSTANCE, Syntax.PerlNG); + return regex.matcher(inputBytes).search(0, inputBytes.length, Option.NONE); + } + + private static String pattern(String left, String assertion, String right) { + return "\\A\\Q" + left + "\\E" + assertion + "\\Q" + right + "\\E\\z"; + } + + private static void assertBoundary(String left, String right) { + String input = left + right; + assertEquals(0, search(pattern(left, "\\b{lb}", right), input)); + assertEquals(-1, search(pattern(left, "\\B{lb}", right), input)); + } + + private static void assertNoBoundary(String left, String right) { + String input = left + right; + assertEquals(-1, search(pattern(left, "\\b{lb}", right), input)); + assertEquals(0, search(pattern(left, "\\B{lb}", right), input)); + } + + @Test + public void implementsUnicodeLineBreakRules() { + assertNoBoundary("", "A"); + assertBoundary("A", ""); + assertBoundary("\u000b", "A"); + assertNoBoundary("\r", "\n"); + assertBoundary("\r", "A"); + assertNoBoundary("A", "\n"); + assertNoBoundary("A", " "); + assertBoundary("\u200b ", "A"); + assertNoBoundary("\u200d", "A"); + assertNoBoundary("A", "\u0308"); + assertBoundary(" ", "\u0308"); + assertNoBoundary("A", "\u2060"); + assertNoBoundary("\u2060", "A"); + assertNoBoundary("\u00a0", "A"); + assertNoBoundary("A", "\u00a0"); + assertBoundary(" ", "\u00a0"); + assertNoBoundary("A ", "]"); + assertNoBoundary("( ", "A"); + assertBoundary(" ", ".5"); + assertNoBoundary("A ", ","); + assertNoBoundary(") ", "\u3041"); + assertNoBoundary("\u2014 ", "\u2014"); + assertBoundary("A ", "B"); + assertNoBoundary("A", "\""); + assertBoundary("A", "\ufffc"); + assertBoundary("\ufffc", "A"); + assertNoBoundary(" -", "A"); + assertNoBoundary("A", "-"); + assertNoBoundary("A", "\u2026"); + assertNoBoundary("A", "1"); + assertNoBoundary("$", "\u4e2d"); + assertNoBoundary("$", "A"); + assertNoBoundary("12,", "3"); + assertNoBoundary("\u1100", "\u1161"); + assertNoBoundary("A", "B"); + assertNoBoundary("\ud804\udc03", "\u1b05"); + assertNoBoundary(".", "A"); + assertNoBoundary("A", "("); + assertNoBoundary("\ud83c\udde6", "\ud83c\udde7"); + assertBoundary("\ud83c\udde6\ud83c\udde7", "\ud83c\udde8"); + assertNoBoundary("\u261d", "\ud83c\udffb"); + assertBoundary("\u4e2d", "\u6587"); + } +} From e03226a8ef00542cd17955a860c6cca7bd254ebb Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:32 +0200 Subject: [PATCH 26/60] feat(regex): pin canonical combining classes (#1031) Squashed integration of PR #1031 at 22b9d3269. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 41 +- ...erate_perl_unicode_combining_class_data.pl | 214 +++++++++ docs/reference/feature-matrix.md | 4 +- .../regex/PerlUnicodeCombiningClassData.java | 454 ++++++++++++++++++ .../runtime/regex/UnicodeResolver.java | 23 + .../PerlUnicodeCombiningClassDataTest.java | 31 ++ ...unicode_canonical_combining_class_pinned.t | 32 ++ 7 files changed, 786 insertions(+), 13 deletions(-) create mode 100644 dev/tools/generate_perl_unicode_combining_class_data.pl create mode 100644 src/main/java/org/perlonjava/runtime/regex/PerlUnicodeCombiningClassData.java create mode 100644 src/test/java/org/perlonjava/runtime/regex/PerlUnicodeCombiningClassDataTest.java create mode 100644 src/test/resources/unit/regex/unicode_canonical_combining_class_pinned.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index a02f5c3578..af08952a18 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,7 +224,7 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1029, stacked +The published integration stack is preserved through draft PR #1030, stacked on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the @@ -232,9 +232,11 @@ first 524 lines of retired Java-only preprocessor code. Every published slice has a warning-free combined `make` checkpoint. PR #1027 adds native sentence boundaries; PR #1028 adds independently validated alpha assertion aliases and native word boundaries. PR #1029 integrates corrected global zero-width `/g` -progression and pinned Perl 5.44 Unicode 17.0 Age properties. The new WIP is -closing combined-corpus property regressions, adding pinned General_Category -sets, and integrating exact native line boundaries. +progression and pinned Perl 5.44 Unicode 17.0 Age properties. PR #1030 adds +binary `ASCII_Hex_Digit` values, pinned General_Category sets, and exact native +line boundaries. The new WIP adds pinned Canonical_Combining_Class sets while +the numeric-escape, Bidi_Class, Decomposition_Type, and vertical-whitespace +slices advance independently. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -299,6 +301,16 @@ pass 205,380/205,380 each on JVM and interpreter. Protected sentence and word chunks remain exact, making the complete generated boundary corpus 239,866/239,866 and current generated evidence 339,075/407,367. +`Canonical_Combining_Class`/`ccc` assignments now resolve every pinned Unicode +17 value and alias, including ordered `Not_Reordered` defaults for unassigned +code points and reserved valid values whose sets are empty. Empty properties +render as valid match-none/match-all classes rather than invalid `[]` syntax. +The focused oracle passes 24/24 on system Perl, JVM, and interpreter. Chunk 01 +passes 33,516/41,843 identically on both execution backends: 2,195 CCC +assertions and 10 already-native line preamble assertions improve over the +31,311 baseline with zero numbered regressions. Current generated evidence is +341,280/407,367. + Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while ordinary and substitution callbacks execute, without misclassifying escaped or @@ -532,6 +544,10 @@ parser slice. - [x] Implemented native Joni line assertions from reproducible pinned Unicode 17.0 data. The focused oracle passes 84/84 and chunks 06–09 pass 205,380/205,380 on both execution backends while sentence/word stay exact. + - [x] Generated pinned Unicode 17.0 Canonical_Combining_Class sets with loose + aliases, ordered missing defaults, reserved empty values, and valid + empty/full rendering. The focused oracle passes 24/24 and chunk 01 reaches + 33,516/41,843 on both backends with zero numbered regressions. - [ ] Close the remaining property failures before marking Phase 3 complete. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) @@ -574,17 +590,20 @@ parser slice. ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1029. Integrate native Joni - unbraced-byte and underscored numeric escape parsing, then rerun the affected - direct/thread files and confirm the last combined-corpus fatal root is closed. +1. Land review-ready PR #1024 and draft PRs #1025–#1030. Publish the pinned CCC + slice, then integrate accepted native numeric-escape commit `f849c2ef9` in a + new WIP PR. Preserve its restored `pat_advanced`/`pat_rt_report` startup and + explicit strict-policy and above-U+10FFFF representation debt. 2. Preserve the now-complete native Joni boundary corpus at 239,866/239,866: sentence chunk 05, line chunks 06–09, and word chunk 10 must remain exact on JVM and interpreter while property and parser work continues. -3. Implement the remaining property clusters in measured order: Block, - Script/Script_Extensions, Numeric_Value, Joining_Group, - break-property values. Preserve pinned Perl 5.44 +3. Integrate the independently generated Bidi_Class and Decomposition_Type + slices, then continue the measured residual order: East_Asian_Width, binary + property values, Block, Script/Script_Extensions, Numeric_Value, + Joining_Group, and break-property values. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. - Close underscored numeric escapes with focused standard-Perl gates. + Integrate native `\v`/`\V` after the numeric Lexer slice and require exact + `reg_posixcc.t` parity. 4. Rerun generated property chunks 01–04 on both backends with the classified 600-second bound and retain complete TAP/JSON. After the two fatal roots and native line boundaries integrate, refresh the complete forced-Joni 80-file diff --git a/dev/tools/generate_perl_unicode_combining_class_data.pl b/dev/tools/generate_perl_unicode_combining_class_data.pl new file mode 100644 index 0000000000..397c7c5f58 --- /dev/null +++ b/dev/tools/generate_perl_unicode_combining_class_data.pl @@ -0,0 +1,214 @@ +#!/usr/bin/env perl +use strict; +use warnings; +use Digest::SHA qw(sha256_hex); + +my $unicode_root = 'perl5/lib/unicore'; +my $expected_unicode_version = '17.0.0'; +my %expected_hash = ( + 'extracted/DCombiningClass.txt' => + '191463abfbd202703c6fd6776a92a23ac44ec65e0476a7f95aa91ca492cef29b', + 'PropValueAliases.txt' => + '670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01', +); + +open my $version_fh, '<', "$unicode_root/version" + or die "Can't read Unicode version: $!\n"; +chomp(my $unicode_version = <$version_fh>); +close $version_fh; +die "Expected Unicode $expected_unicode_version, found $unicode_version\n" + unless $unicode_version eq $expected_unicode_version; + +sub read_pinned_file { + my ($relative) = @_; + my $path = "$unicode_root/$relative"; + open my $fh, '<:raw', $path or die "Can't read $path: $!\n"; + local $/; + my $text = <$fh>; + close $fh; + my $actual = sha256_hex($text); + die "$path SHA-256 mismatch: expected $expected_hash{$relative}, found $actual\n" + unless $actual eq $expected_hash{$relative}; + return $text; +} + +sub normalized_alias { + my ($alias) = @_; + $alias = lc $alias; + $alias =~ s/[\s_+\-]//g; + $alias =~ s/^0+(?=\d)// if $alias =~ /^\d+$/; + return $alias; +} + +my $class_text = read_pinned_file('extracted/DCombiningClass.txt'); +die "Combining class data is not Unicode $expected_unicode_version\n" + unless $class_text =~ /^# DerivedCombiningClass-\Q$expected_unicode_version\E\.txt/m; + +my (@values, %value_index, @ranges); +for my $line (split /\n/, $class_text) { + next unless $line =~ /^([0-9A-F]+)(?:\.\.([0-9A-F]+))?\s*;\s*([0-9]+)/; + my ($start, $end, $value) = + (hex($1), defined($2) ? hex($2) : hex($1), int($3)); + if (!exists $value_index{$value}) { + $value_index{$value} = scalar @values; + push @values, $value; + } + my $index = $value_index{$value}; + if (@ranges && $ranges[-1][2] == $index && $ranges[-1][1] + 1 == $start) { + $ranges[-1][1] = $end; + } else { + push @ranges, [$start, $end, $index]; + } +} +die "No Canonical_Combining_Class ranges found\n" unless @ranges; +die "Unexpected Canonical_Combining_Class default\n" + unless $class_text =~ /^# \@missing:\s*0000\.\.10FFFF;\s*Not_Reordered\s*$/m; + +my $alias_text = read_pinned_file('PropValueAliases.txt'); +die "Property value aliases are not Unicode $expected_unicode_version\n" + unless $alias_text =~ /^# PropertyValueAliases-\Q$expected_unicode_version\E\.txt/m; +my %aliases; +for my $line (split /\n/, $alias_text) { + $line =~ s/\s*#.*$//; + next unless $line =~ /\S/; + my @fields = map { + my $field = $_; + $field =~ s/^\s+|\s+$//g; + $field; + } split /;/, $line; + next unless @fields >= 4 && lc($fields[0]) eq 'ccc'; + my $canonical = int($fields[1]); + if (!exists $value_index{$canonical}) { + $value_index{$canonical} = scalar @values; + push @values, $canonical; + } + for my $alias (@fields[1 .. $#fields]) { + next unless length $alias; + my $normalized = normalized_alias($alias); + if (exists $aliases{$normalized} && $aliases{$normalized} != $canonical) { + die "Conflicting combining-class alias $alias\n"; + } + $aliases{$normalized} = $canonical; + } +} +for my $value (@values) { + $aliases{normalized_alias($value)} //= $value; +} + +print <<'HEADER'; +/* + * Generated from Perl 5.44's Unicode Character Database. Do not edit manually. + * + * Unicode data source copyright: + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in + * the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + */ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; + +import java.util.HashMap; +import java.util.Map; + +final class PerlUnicodeCombiningClassData { +HEADER + +print " static final String UNICODE_VERSION = \"$unicode_version\";\n"; +print " private static final int DEFAULT_VALUE_INDEX = $value_index{0};\n"; +print " private static final int[] VALUES = {\n "; +print join(', ', @values); +print "\n };\n\n"; + +my $range_chunk_size = 400; +my $range_chunk_count = int((@ranges + $range_chunk_size - 1) / $range_chunk_size); +for my $chunk (0 .. $range_chunk_count - 1) { + my $first = $chunk * $range_chunk_size; + my $last = $first + $range_chunk_size - 1; + $last = $#ranges if $last > $#ranges; + print " private static int[] rangeChunk$chunk() {\n"; + print " return new int[] {\n"; + for (my $i = $first; $i <= $last; $i += 4) { + my $end = $i + 3 < $last ? $i + 3 : $last; + print " ", join(', ', map { + sprintf '0x%X, 0x%X, %d', @{$ranges[$_]}[0, 1, 2] + } $i .. $end), ",\n"; + } + print " };\n"; + print " }\n\n"; +} +print " private static final int[][] RANGE_CHUNKS = {\n "; +print join(', ', map { "rangeChunk$_()" } 0 .. $range_chunk_count - 1); +print "\n };\n\n"; + +print " private static final String[] ALIASES = {\n"; +my @alias_names = sort keys %aliases; +for (my $i = 0; $i < @alias_names; $i += 4) { + my $end = $i + 3 < $#alias_names ? $i + 3 : $#alias_names; + print " ", join(', ', map { + my $name = $alias_names[$_]; + qq{"$name", "} . $aliases{$name} . qq{"} + } $i .. $end), ",\n"; +} +print <<'FOOTER'; + }; + + private static final UnicodeSet[] SETS = buildSets(); + private static final Map ALIAS_INDEX = buildAliasIndex(); + + static UnicodeSet resolve(String value) { + Integer index = ALIAS_INDEX.get(normalizeValue(value)); + return index == null ? null : SETS[index]; + } + + private static UnicodeSet[] buildSets() { + UnicodeSet[] sets = new UnicodeSet[VALUES.length]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + UnicodeSet covered = new UnicodeSet(); + for (int[] ranges : RANGE_CHUNKS) { + for (int i = 0; i < ranges.length; i += 3) { + sets[ranges[i + 2]].add(ranges[i], ranges[i + 1]); + covered.add(ranges[i], ranges[i + 1]); + } + } + sets[DEFAULT_VALUE_INDEX].addAll( + new UnicodeSet(0, 0x10FFFF).removeAll(covered)); + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + + private static Map buildAliasIndex() { + Map valueIndexes = new HashMap<>(); + for (int i = 0; i < VALUES.length; i++) valueIndexes.put(VALUES[i], i); + Map indexes = new HashMap<>(); + for (int i = 0; i < ALIASES.length; i += 2) { + indexes.put(ALIASES[i], valueIndexes.get(Integer.parseInt(ALIASES[i + 1]))); + } + return Map.copyOf(indexes); + } + + private static String normalizeValue(String value) { + String normalized = value.trim(); + if (normalized.startsWith(":\\A") && normalized.endsWith("\\z:") + && normalized.length() > 6) { + normalized = normalized.substring(3, normalized.length() - 3); + } + StringBuilder loose = new StringBuilder(normalized.length()); + boolean numeric = true; + for (int i = 0; i < normalized.length(); i++) { + char ch = normalized.charAt(i); + if (Character.isWhitespace(ch) || ch == '-' || ch == '_' || ch == '+') continue; + if (!Character.isDigit(ch)) numeric = false; + loose.append(Character.toLowerCase(ch)); + } + if (!numeric) return loose.toString(); + int first = 0; + while (first + 1 < loose.length() && loose.charAt(first) == '0') first++; + return loose.substring(first); + } + + private PerlUnicodeCombiningClassData() { + } +} +FOOTER diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index b912639f82..6a85ed5d1c 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -378,7 +378,7 @@ my @copy = @{$z}; # ERROR - ✅ **Backreferences to Named Groups**: Using `\k` or `\g{name}` for backreferences to named groups is supported. - ✅ **Relative Backreferences**: Using `\g{-n}` for relative backreferences. - ✅ **Basic Unicode Properties**: Common `\p{...}` and `\P{...}` forms such as `\p{L}` execute through Joni. -- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, and all General_Category assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. +- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, and Canonical_Combining_Class assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, and reserved empty values; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. - ✅ **Possessive Quantifiers**: Quantifiers like `*+`, `++`, `?+`, and `{n,m}+`, which disable backtracking, are supported. - ✅ **Atomic Grouping**: Use of `(?>...)` for atomic groups is supported. - ✅ **`\K` assertion**: Keep left — in `s///`, text before `\K` is preserved; match variables reflect only the portion after `\K`. @@ -397,7 +397,7 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 339,075/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 341,280/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. - ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. - ✅ **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter. - ✅ **Unicode Word Boundaries**: Native `\b{wb}`/`\B{wb}` implement WB1–WB16 and WB999 from reproducible Perl 5.44 Unicode 17.0 word-break and extended-pictographic tables. Authoritative chunk 10 passes 19,510/19,510 identically on JVM and interpreter. diff --git a/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeCombiningClassData.java b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeCombiningClassData.java new file mode 100644 index 0000000000..3ad2885006 --- /dev/null +++ b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeCombiningClassData.java @@ -0,0 +1,454 @@ +/* + * Generated from Perl 5.44's Unicode Character Database. Do not edit manually. + * + * Unicode data source copyright: + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in + * the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + */ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; + +import java.util.HashMap; +import java.util.Map; + +final class PerlUnicodeCombiningClassData { + static final String UNICODE_VERSION = "17.0.0"; + private static final int DEFAULT_VALUE_INDEX = 0; + private static final int[] VALUES = { + 0, 1, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 84, 91, 103, 107, 118, 122, 129, 130, 132, 202, 214, 216, 218, 220, 222, 224, 226, 228, 230, 232, 233, 234, 240, 133, 200 + }; + + private static int[] rangeChunk0() { + return new int[] { + 0x0, 0x2FF, 0, 0x34F, 0x34F, 0, 0x370, 0x377, 0, 0x37A, 0x37F, 0, + 0x384, 0x38A, 0, 0x38C, 0x38C, 0, 0x38E, 0x3A1, 0, 0x3A3, 0x482, 0, + 0x488, 0x52F, 0, 0x531, 0x556, 0, 0x559, 0x58A, 0, 0x58D, 0x58F, 0, + 0x5BE, 0x5BE, 0, 0x5C0, 0x5C0, 0, 0x5C3, 0x5C3, 0, 0x5C6, 0x5C6, 0, + 0x5D0, 0x5EA, 0, 0x5EF, 0x5F4, 0, 0x600, 0x60F, 0, 0x61B, 0x64A, 0, + 0x660, 0x66F, 0, 0x671, 0x6D5, 0, 0x6DD, 0x6DE, 0, 0x6E5, 0x6E6, 0, + 0x6E9, 0x6E9, 0, 0x6EE, 0x70D, 0, 0x70F, 0x710, 0, 0x712, 0x72F, 0, + 0x74D, 0x7B1, 0, 0x7C0, 0x7EA, 0, 0x7F4, 0x7FA, 0, 0x7FE, 0x815, 0, + 0x81A, 0x81A, 0, 0x824, 0x824, 0, 0x828, 0x828, 0, 0x830, 0x83E, 0, + 0x840, 0x858, 0, 0x85E, 0x85E, 0, 0x860, 0x86A, 0, 0x870, 0x891, 0, + 0x8A0, 0x8C9, 0, 0x8E2, 0x8E2, 0, 0x900, 0x93B, 0, 0x93D, 0x94C, 0, + 0x94E, 0x950, 0, 0x955, 0x983, 0, 0x985, 0x98C, 0, 0x98F, 0x990, 0, + 0x993, 0x9A8, 0, 0x9AA, 0x9B0, 0, 0x9B2, 0x9B2, 0, 0x9B6, 0x9B9, 0, + 0x9BD, 0x9C4, 0, 0x9C7, 0x9C8, 0, 0x9CB, 0x9CC, 0, 0x9CE, 0x9CE, 0, + 0x9D7, 0x9D7, 0, 0x9DC, 0x9DD, 0, 0x9DF, 0x9E3, 0, 0x9E6, 0x9FD, 0, + 0xA01, 0xA03, 0, 0xA05, 0xA0A, 0, 0xA0F, 0xA10, 0, 0xA13, 0xA28, 0, + 0xA2A, 0xA30, 0, 0xA32, 0xA33, 0, 0xA35, 0xA36, 0, 0xA38, 0xA39, 0, + 0xA3E, 0xA42, 0, 0xA47, 0xA48, 0, 0xA4B, 0xA4C, 0, 0xA51, 0xA51, 0, + 0xA59, 0xA5C, 0, 0xA5E, 0xA5E, 0, 0xA66, 0xA76, 0, 0xA81, 0xA83, 0, + 0xA85, 0xA8D, 0, 0xA8F, 0xA91, 0, 0xA93, 0xAA8, 0, 0xAAA, 0xAB0, 0, + 0xAB2, 0xAB3, 0, 0xAB5, 0xAB9, 0, 0xABD, 0xAC5, 0, 0xAC7, 0xAC9, 0, + 0xACB, 0xACC, 0, 0xAD0, 0xAD0, 0, 0xAE0, 0xAE3, 0, 0xAE6, 0xAF1, 0, + 0xAF9, 0xAFF, 0, 0xB01, 0xB03, 0, 0xB05, 0xB0C, 0, 0xB0F, 0xB10, 0, + 0xB13, 0xB28, 0, 0xB2A, 0xB30, 0, 0xB32, 0xB33, 0, 0xB35, 0xB39, 0, + 0xB3D, 0xB44, 0, 0xB47, 0xB48, 0, 0xB4B, 0xB4C, 0, 0xB55, 0xB57, 0, + 0xB5C, 0xB5D, 0, 0xB5F, 0xB63, 0, 0xB66, 0xB77, 0, 0xB82, 0xB83, 0, + 0xB85, 0xB8A, 0, 0xB8E, 0xB90, 0, 0xB92, 0xB95, 0, 0xB99, 0xB9A, 0, + 0xB9C, 0xB9C, 0, 0xB9E, 0xB9F, 0, 0xBA3, 0xBA4, 0, 0xBA8, 0xBAA, 0, + 0xBAE, 0xBB9, 0, 0xBBE, 0xBC2, 0, 0xBC6, 0xBC8, 0, 0xBCA, 0xBCC, 0, + 0xBD0, 0xBD0, 0, 0xBD7, 0xBD7, 0, 0xBE6, 0xBFA, 0, 0xC00, 0xC0C, 0, + 0xC0E, 0xC10, 0, 0xC12, 0xC28, 0, 0xC2A, 0xC39, 0, 0xC3D, 0xC44, 0, + 0xC46, 0xC48, 0, 0xC4A, 0xC4C, 0, 0xC58, 0xC5A, 0, 0xC5C, 0xC5D, 0, + 0xC60, 0xC63, 0, 0xC66, 0xC6F, 0, 0xC77, 0xC8C, 0, 0xC8E, 0xC90, 0, + 0xC92, 0xCA8, 0, 0xCAA, 0xCB3, 0, 0xCB5, 0xCB9, 0, 0xCBD, 0xCC4, 0, + 0xCC6, 0xCC8, 0, 0xCCA, 0xCCC, 0, 0xCD5, 0xCD6, 0, 0xCDC, 0xCDE, 0, + 0xCE0, 0xCE3, 0, 0xCE6, 0xCEF, 0, 0xCF1, 0xCF3, 0, 0xD00, 0xD0C, 0, + 0xD0E, 0xD10, 0, 0xD12, 0xD3A, 0, 0xD3D, 0xD44, 0, 0xD46, 0xD48, 0, + 0xD4A, 0xD4C, 0, 0xD4E, 0xD4F, 0, 0xD54, 0xD63, 0, 0xD66, 0xD7F, 0, + 0xD81, 0xD83, 0, 0xD85, 0xD96, 0, 0xD9A, 0xDB1, 0, 0xDB3, 0xDBB, 0, + 0xDBD, 0xDBD, 0, 0xDC0, 0xDC6, 0, 0xDCF, 0xDD4, 0, 0xDD6, 0xDD6, 0, + 0xDD8, 0xDDF, 0, 0xDE6, 0xDEF, 0, 0xDF2, 0xDF4, 0, 0xE01, 0xE37, 0, + 0xE3F, 0xE47, 0, 0xE4C, 0xE5B, 0, 0xE81, 0xE82, 0, 0xE84, 0xE84, 0, + 0xE86, 0xE8A, 0, 0xE8C, 0xEA3, 0, 0xEA5, 0xEA5, 0, 0xEA7, 0xEB7, 0, + 0xEBB, 0xEBD, 0, 0xEC0, 0xEC4, 0, 0xEC6, 0xEC6, 0, 0xECC, 0xECE, 0, + 0xED0, 0xED9, 0, 0xEDC, 0xEDF, 0, 0xF00, 0xF17, 0, 0xF1A, 0xF34, 0, + 0xF36, 0xF36, 0, 0xF38, 0xF38, 0, 0xF3A, 0xF47, 0, 0xF49, 0xF6C, 0, + 0xF73, 0xF73, 0, 0xF75, 0xF79, 0, 0xF7E, 0xF7F, 0, 0xF81, 0xF81, 0, + 0xF85, 0xF85, 0, 0xF88, 0xF97, 0, 0xF99, 0xFBC, 0, 0xFBE, 0xFC5, 0, + 0xFC7, 0xFCC, 0, 0xFCE, 0xFDA, 0, 0x1000, 0x1036, 0, 0x1038, 0x1038, 0, + 0x103B, 0x108C, 0, 0x108E, 0x10C5, 0, 0x10C7, 0x10C7, 0, 0x10CD, 0x10CD, 0, + 0x10D0, 0x1248, 0, 0x124A, 0x124D, 0, 0x1250, 0x1256, 0, 0x1258, 0x1258, 0, + 0x125A, 0x125D, 0, 0x1260, 0x1288, 0, 0x128A, 0x128D, 0, 0x1290, 0x12B0, 0, + 0x12B2, 0x12B5, 0, 0x12B8, 0x12BE, 0, 0x12C0, 0x12C0, 0, 0x12C2, 0x12C5, 0, + 0x12C8, 0x12D6, 0, 0x12D8, 0x1310, 0, 0x1312, 0x1315, 0, 0x1318, 0x135A, 0, + 0x1360, 0x137C, 0, 0x1380, 0x1399, 0, 0x13A0, 0x13F5, 0, 0x13F8, 0x13FD, 0, + 0x1400, 0x169C, 0, 0x16A0, 0x16F8, 0, 0x1700, 0x1713, 0, 0x171F, 0x1733, 0, + 0x1735, 0x1736, 0, 0x1740, 0x1753, 0, 0x1760, 0x176C, 0, 0x176E, 0x1770, 0, + 0x1772, 0x1773, 0, 0x1780, 0x17D1, 0, 0x17D3, 0x17DC, 0, 0x17E0, 0x17E9, 0, + 0x17F0, 0x17F9, 0, 0x1800, 0x1819, 0, 0x1820, 0x1878, 0, 0x1880, 0x18A8, 0, + 0x18AA, 0x18AA, 0, 0x18B0, 0x18F5, 0, 0x1900, 0x191E, 0, 0x1920, 0x192B, 0, + 0x1930, 0x1938, 0, 0x1940, 0x1940, 0, 0x1944, 0x196D, 0, 0x1970, 0x1974, 0, + 0x1980, 0x19AB, 0, 0x19B0, 0x19C9, 0, 0x19D0, 0x19DA, 0, 0x19DE, 0x1A16, 0, + 0x1A19, 0x1A1B, 0, 0x1A1E, 0x1A5E, 0, 0x1A61, 0x1A74, 0, 0x1A80, 0x1A89, 0, + 0x1A90, 0x1A99, 0, 0x1AA0, 0x1AAD, 0, 0x1ABE, 0x1ABE, 0, 0x1B00, 0x1B33, 0, + 0x1B35, 0x1B43, 0, 0x1B45, 0x1B4C, 0, 0x1B4E, 0x1B6A, 0, 0x1B74, 0x1BA9, 0, + 0x1BAC, 0x1BE5, 0, 0x1BE7, 0x1BF1, 0, 0x1BFC, 0x1C36, 0, 0x1C3B, 0x1C49, 0, + 0x1C4D, 0x1C8A, 0, 0x1C90, 0x1CBA, 0, 0x1CBD, 0x1CC7, 0, 0x1CD3, 0x1CD3, 0, + 0x1CE1, 0x1CE1, 0, 0x1CE9, 0x1CEC, 0, 0x1CEE, 0x1CF3, 0, 0x1CF5, 0x1CF7, 0, + 0x1CFA, 0x1CFA, 0, 0x1D00, 0x1DBF, 0, 0x1E00, 0x1F15, 0, 0x1F18, 0x1F1D, 0, + 0x1F20, 0x1F45, 0, 0x1F48, 0x1F4D, 0, 0x1F50, 0x1F57, 0, 0x1F59, 0x1F59, 0, + 0x1F5B, 0x1F5B, 0, 0x1F5D, 0x1F5D, 0, 0x1F5F, 0x1F7D, 0, 0x1F80, 0x1FB4, 0, + 0x1FB6, 0x1FC4, 0, 0x1FC6, 0x1FD3, 0, 0x1FD6, 0x1FDB, 0, 0x1FDD, 0x1FEF, 0, + 0x1FF2, 0x1FF4, 0, 0x1FF6, 0x1FFE, 0, 0x2000, 0x2064, 0, 0x2066, 0x2071, 0, + 0x2074, 0x208E, 0, 0x2090, 0x209C, 0, 0x20A0, 0x20C1, 0, 0x20DD, 0x20E0, 0, + 0x20E2, 0x20E4, 0, 0x2100, 0x218B, 0, 0x2190, 0x2429, 0, 0x2440, 0x244A, 0, + 0x2460, 0x2B73, 0, 0x2B76, 0x2CEE, 0, 0x2CF2, 0x2CF3, 0, 0x2CF9, 0x2D25, 0, + 0x2D27, 0x2D27, 0, 0x2D2D, 0x2D2D, 0, 0x2D30, 0x2D67, 0, 0x2D6F, 0x2D70, 0, + 0x2D80, 0x2D96, 0, 0x2DA0, 0x2DA6, 0, 0x2DA8, 0x2DAE, 0, 0x2DB0, 0x2DB6, 0, + 0x2DB8, 0x2DBE, 0, 0x2DC0, 0x2DC6, 0, 0x2DC8, 0x2DCE, 0, 0x2DD0, 0x2DD6, 0, + 0x2DD8, 0x2DDE, 0, 0x2E00, 0x2E5D, 0, 0x2E80, 0x2E99, 0, 0x2E9B, 0x2EF3, 0, + 0x2F00, 0x2FD5, 0, 0x2FF0, 0x3029, 0, 0x3030, 0x303F, 0, 0x3041, 0x3096, 0, + 0x309B, 0x30FF, 0, 0x3105, 0x312F, 0, 0x3131, 0x318E, 0, 0x3190, 0x31E5, 0, + 0x31EF, 0x321E, 0, 0x3220, 0xA48C, 0, 0xA490, 0xA4C6, 0, 0xA4D0, 0xA62B, 0, + 0xA640, 0xA66E, 0, 0xA670, 0xA673, 0, 0xA67E, 0xA69D, 0, 0xA6A0, 0xA6EF, 0, + 0xA6F2, 0xA6F7, 0, 0xA700, 0xA7DC, 0, 0xA7F1, 0xA805, 0, 0xA807, 0xA82B, 0, + 0xA830, 0xA839, 0, 0xA840, 0xA877, 0, 0xA880, 0xA8C3, 0, 0xA8C5, 0xA8C5, 0, + 0xA8CE, 0xA8D9, 0, 0xA8F2, 0xA92A, 0, 0xA92E, 0xA952, 0, 0xA95F, 0xA97C, 0, + 0xA980, 0xA9B2, 0, 0xA9B4, 0xA9BF, 0, 0xA9C1, 0xA9CD, 0, 0xA9CF, 0xA9D9, 0, + 0xA9DE, 0xA9FE, 0, 0xAA00, 0xAA36, 0, 0xAA40, 0xAA4D, 0, 0xAA50, 0xAA59, 0, + 0xAA5C, 0xAAAF, 0, 0xAAB1, 0xAAB1, 0, 0xAAB5, 0xAAB6, 0, 0xAAB9, 0xAABD, 0, + 0xAAC0, 0xAAC0, 0, 0xAAC2, 0xAAC2, 0, 0xAADB, 0xAAF5, 0, 0xAB01, 0xAB06, 0, + 0xAB09, 0xAB0E, 0, 0xAB11, 0xAB16, 0, 0xAB20, 0xAB26, 0, 0xAB28, 0xAB2E, 0, + 0xAB30, 0xAB6B, 0, 0xAB70, 0xABEC, 0, 0xABF0, 0xABF9, 0, 0xAC00, 0xD7A3, 0, + 0xD7B0, 0xD7C6, 0, 0xD7CB, 0xD7FB, 0, 0xE000, 0xFA6D, 0, 0xFA70, 0xFAD9, 0, + 0xFB00, 0xFB06, 0, 0xFB13, 0xFB17, 0, 0xFB1D, 0xFB1D, 0, 0xFB1F, 0xFB36, 0, + 0xFB38, 0xFB3C, 0, 0xFB3E, 0xFB3E, 0, 0xFB40, 0xFB41, 0, 0xFB43, 0xFB44, 0, + 0xFB46, 0xFDCF, 0, 0xFDF0, 0xFE19, 0, 0xFE30, 0xFE52, 0, 0xFE54, 0xFE66, 0, + 0xFE68, 0xFE6B, 0, 0xFE70, 0xFE74, 0, 0xFE76, 0xFEFC, 0, 0xFEFF, 0xFEFF, 0, + 0xFF01, 0xFFBE, 0, 0xFFC2, 0xFFC7, 0, 0xFFCA, 0xFFCF, 0, 0xFFD2, 0xFFD7, 0, + 0xFFDA, 0xFFDC, 0, 0xFFE0, 0xFFE6, 0, 0xFFE8, 0xFFEE, 0, 0xFFF9, 0xFFFD, 0, + }; + } + + private static int[] rangeChunk1() { + return new int[] { + 0x10000, 0x1000B, 0, 0x1000D, 0x10026, 0, 0x10028, 0x1003A, 0, 0x1003C, 0x1003D, 0, + 0x1003F, 0x1004D, 0, 0x10050, 0x1005D, 0, 0x10080, 0x100FA, 0, 0x10100, 0x10102, 0, + 0x10107, 0x10133, 0, 0x10137, 0x1018E, 0, 0x10190, 0x1019C, 0, 0x101A0, 0x101A0, 0, + 0x101D0, 0x101FC, 0, 0x10280, 0x1029C, 0, 0x102A0, 0x102D0, 0, 0x102E1, 0x102FB, 0, + 0x10300, 0x10323, 0, 0x1032D, 0x1034A, 0, 0x10350, 0x10375, 0, 0x10380, 0x1039D, 0, + 0x1039F, 0x103C3, 0, 0x103C8, 0x103D5, 0, 0x10400, 0x1049D, 0, 0x104A0, 0x104A9, 0, + 0x104B0, 0x104D3, 0, 0x104D8, 0x104FB, 0, 0x10500, 0x10527, 0, 0x10530, 0x10563, 0, + 0x1056F, 0x1057A, 0, 0x1057C, 0x1058A, 0, 0x1058C, 0x10592, 0, 0x10594, 0x10595, 0, + 0x10597, 0x105A1, 0, 0x105A3, 0x105B1, 0, 0x105B3, 0x105B9, 0, 0x105BB, 0x105BC, 0, + 0x105C0, 0x105F3, 0, 0x10600, 0x10736, 0, 0x10740, 0x10755, 0, 0x10760, 0x10767, 0, + 0x10780, 0x10785, 0, 0x10787, 0x107B0, 0, 0x107B2, 0x107BA, 0, 0x10800, 0x10805, 0, + 0x10808, 0x10808, 0, 0x1080A, 0x10835, 0, 0x10837, 0x10838, 0, 0x1083C, 0x1083C, 0, + 0x1083F, 0x10855, 0, 0x10857, 0x1089E, 0, 0x108A7, 0x108AF, 0, 0x108E0, 0x108F2, 0, + 0x108F4, 0x108F5, 0, 0x108FB, 0x1091B, 0, 0x1091F, 0x10939, 0, 0x1093F, 0x10959, 0, + 0x10980, 0x109B7, 0, 0x109BC, 0x109CF, 0, 0x109D2, 0x10A03, 0, 0x10A05, 0x10A06, 0, + 0x10A0C, 0x10A0C, 0, 0x10A0E, 0x10A0E, 0, 0x10A10, 0x10A13, 0, 0x10A15, 0x10A17, 0, + 0x10A19, 0x10A35, 0, 0x10A40, 0x10A48, 0, 0x10A50, 0x10A58, 0, 0x10A60, 0x10A9F, 0, + 0x10AC0, 0x10AE4, 0, 0x10AEB, 0x10AF6, 0, 0x10B00, 0x10B35, 0, 0x10B39, 0x10B55, 0, + 0x10B58, 0x10B72, 0, 0x10B78, 0x10B91, 0, 0x10B99, 0x10B9C, 0, 0x10BA9, 0x10BAF, 0, + 0x10C00, 0x10C48, 0, 0x10C80, 0x10CB2, 0, 0x10CC0, 0x10CF2, 0, 0x10CFA, 0x10D23, 0, + 0x10D30, 0x10D39, 0, 0x10D40, 0x10D65, 0, 0x10D6E, 0x10D85, 0, 0x10D8E, 0x10D8F, 0, + 0x10E60, 0x10E7E, 0, 0x10E80, 0x10EA9, 0, 0x10EAD, 0x10EAD, 0, 0x10EB0, 0x10EB1, 0, + 0x10EC2, 0x10EC7, 0, 0x10ED0, 0x10ED8, 0, 0x10EFC, 0x10EFC, 0, 0x10F00, 0x10F27, 0, + 0x10F30, 0x10F45, 0, 0x10F51, 0x10F59, 0, 0x10F70, 0x10F81, 0, 0x10F86, 0x10F89, 0, + 0x10FB0, 0x10FCB, 0, 0x10FE0, 0x10FF6, 0, 0x11000, 0x11045, 0, 0x11047, 0x1104D, 0, + 0x11052, 0x1106F, 0, 0x11071, 0x11075, 0, 0x11080, 0x110B8, 0, 0x110BB, 0x110C2, 0, + 0x110CD, 0x110CD, 0, 0x110D0, 0x110E8, 0, 0x110F0, 0x110F9, 0, 0x11103, 0x11132, 0, + 0x11136, 0x11147, 0, 0x11150, 0x11172, 0, 0x11174, 0x11176, 0, 0x11180, 0x111BF, 0, + 0x111C1, 0x111C9, 0, 0x111CB, 0x111DF, 0, 0x111E1, 0x111F4, 0, 0x11200, 0x11211, 0, + 0x11213, 0x11234, 0, 0x11237, 0x11241, 0, 0x11280, 0x11286, 0, 0x11288, 0x11288, 0, + 0x1128A, 0x1128D, 0, 0x1128F, 0x1129D, 0, 0x1129F, 0x112A9, 0, 0x112B0, 0x112E8, 0, + 0x112F0, 0x112F9, 0, 0x11300, 0x11303, 0, 0x11305, 0x1130C, 0, 0x1130F, 0x11310, 0, + 0x11313, 0x11328, 0, 0x1132A, 0x11330, 0, 0x11332, 0x11333, 0, 0x11335, 0x11339, 0, + 0x1133D, 0x11344, 0, 0x11347, 0x11348, 0, 0x1134B, 0x1134C, 0, 0x11350, 0x11350, 0, + 0x11357, 0x11357, 0, 0x1135D, 0x11363, 0, 0x11380, 0x11389, 0, 0x1138B, 0x1138B, 0, + 0x1138E, 0x1138E, 0, 0x11390, 0x113B5, 0, 0x113B7, 0x113C0, 0, 0x113C2, 0x113C2, 0, + 0x113C5, 0x113C5, 0, 0x113C7, 0x113CA, 0, 0x113CC, 0x113CD, 0, 0x113D1, 0x113D5, 0, + 0x113D7, 0x113D8, 0, 0x113E1, 0x113E2, 0, 0x11400, 0x11441, 0, 0x11443, 0x11445, 0, + 0x11447, 0x1145B, 0, 0x1145D, 0x1145D, 0, 0x1145F, 0x11461, 0, 0x11480, 0x114C1, 0, + 0x114C4, 0x114C7, 0, 0x114D0, 0x114D9, 0, 0x11580, 0x115B5, 0, 0x115B8, 0x115BE, 0, + 0x115C1, 0x115DD, 0, 0x11600, 0x1163E, 0, 0x11640, 0x11644, 0, 0x11650, 0x11659, 0, + 0x11660, 0x1166C, 0, 0x11680, 0x116B5, 0, 0x116B8, 0x116B9, 0, 0x116C0, 0x116C9, 0, + 0x116D0, 0x116E3, 0, 0x11700, 0x1171A, 0, 0x1171D, 0x1172A, 0, 0x11730, 0x11746, 0, + 0x11800, 0x11838, 0, 0x1183B, 0x1183B, 0, 0x118A0, 0x118F2, 0, 0x118FF, 0x11906, 0, + 0x11909, 0x11909, 0, 0x1190C, 0x11913, 0, 0x11915, 0x11916, 0, 0x11918, 0x11935, 0, + 0x11937, 0x11938, 0, 0x1193B, 0x1193C, 0, 0x1193F, 0x11942, 0, 0x11944, 0x11946, 0, + 0x11950, 0x11959, 0, 0x119A0, 0x119A7, 0, 0x119AA, 0x119D7, 0, 0x119DA, 0x119DF, 0, + 0x119E1, 0x119E4, 0, 0x11A00, 0x11A33, 0, 0x11A35, 0x11A46, 0, 0x11A50, 0x11A98, 0, + 0x11A9A, 0x11AA2, 0, 0x11AB0, 0x11AF8, 0, 0x11B00, 0x11B09, 0, 0x11B60, 0x11B67, 0, + 0x11BC0, 0x11BE1, 0, 0x11BF0, 0x11BF9, 0, 0x11C00, 0x11C08, 0, 0x11C0A, 0x11C36, 0, + 0x11C38, 0x11C3E, 0, 0x11C40, 0x11C45, 0, 0x11C50, 0x11C6C, 0, 0x11C70, 0x11C8F, 0, + 0x11C92, 0x11CA7, 0, 0x11CA9, 0x11CB6, 0, 0x11D00, 0x11D06, 0, 0x11D08, 0x11D09, 0, + 0x11D0B, 0x11D36, 0, 0x11D3A, 0x11D3A, 0, 0x11D3C, 0x11D3D, 0, 0x11D3F, 0x11D41, 0, + 0x11D43, 0x11D43, 0, 0x11D46, 0x11D47, 0, 0x11D50, 0x11D59, 0, 0x11D60, 0x11D65, 0, + 0x11D67, 0x11D68, 0, 0x11D6A, 0x11D8E, 0, 0x11D90, 0x11D91, 0, 0x11D93, 0x11D96, 0, + 0x11D98, 0x11D98, 0, 0x11DA0, 0x11DA9, 0, 0x11DB0, 0x11DDB, 0, 0x11DE0, 0x11DE9, 0, + 0x11EE0, 0x11EF8, 0, 0x11F00, 0x11F10, 0, 0x11F12, 0x11F3A, 0, 0x11F3E, 0x11F40, 0, + 0x11F43, 0x11F5A, 0, 0x11FB0, 0x11FB0, 0, 0x11FC0, 0x11FF1, 0, 0x11FFF, 0x12399, 0, + 0x12400, 0x1246E, 0, 0x12470, 0x12474, 0, 0x12480, 0x12543, 0, 0x12F90, 0x12FF2, 0, + 0x13000, 0x13455, 0, 0x13460, 0x143FA, 0, 0x14400, 0x14646, 0, 0x16100, 0x1612E, 0, + 0x16130, 0x16139, 0, 0x16800, 0x16A38, 0, 0x16A40, 0x16A5E, 0, 0x16A60, 0x16A69, 0, + 0x16A6E, 0x16ABE, 0, 0x16AC0, 0x16AC9, 0, 0x16AD0, 0x16AED, 0, 0x16AF5, 0x16AF5, 0, + 0x16B00, 0x16B2F, 0, 0x16B37, 0x16B45, 0, 0x16B50, 0x16B59, 0, 0x16B5B, 0x16B61, 0, + 0x16B63, 0x16B77, 0, 0x16B7D, 0x16B8F, 0, 0x16D40, 0x16D79, 0, 0x16E40, 0x16E9A, 0, + 0x16EA0, 0x16EB8, 0, 0x16EBB, 0x16ED3, 0, 0x16F00, 0x16F4A, 0, 0x16F4F, 0x16F87, 0, + 0x16F8F, 0x16F9F, 0, 0x16FE0, 0x16FE4, 0, 0x16FF2, 0x16FF6, 0, 0x17000, 0x18CD5, 0, + 0x18CFF, 0x18D1E, 0, 0x18D80, 0x18DF2, 0, 0x1AFF0, 0x1AFF3, 0, 0x1AFF5, 0x1AFFB, 0, + 0x1AFFD, 0x1AFFE, 0, 0x1B000, 0x1B122, 0, 0x1B132, 0x1B132, 0, 0x1B150, 0x1B152, 0, + 0x1B155, 0x1B155, 0, 0x1B164, 0x1B167, 0, 0x1B170, 0x1B2FB, 0, 0x1BC00, 0x1BC6A, 0, + 0x1BC70, 0x1BC7C, 0, 0x1BC80, 0x1BC88, 0, 0x1BC90, 0x1BC99, 0, 0x1BC9C, 0x1BC9D, 0, + 0x1BC9F, 0x1BCA3, 0, 0x1CC00, 0x1CCFC, 0, 0x1CD00, 0x1CEB3, 0, 0x1CEBA, 0x1CED0, 0, + 0x1CEE0, 0x1CEF0, 0, 0x1CF00, 0x1CF2D, 0, 0x1CF30, 0x1CF46, 0, 0x1CF50, 0x1CFC3, 0, + 0x1D000, 0x1D0F5, 0, 0x1D100, 0x1D126, 0, 0x1D129, 0x1D164, 0, 0x1D16A, 0x1D16C, 0, + 0x1D173, 0x1D17A, 0, 0x1D183, 0x1D184, 0, 0x1D18C, 0x1D1A9, 0, 0x1D1AE, 0x1D1EA, 0, + 0x1D200, 0x1D241, 0, 0x1D245, 0x1D245, 0, 0x1D2C0, 0x1D2D3, 0, 0x1D2E0, 0x1D2F3, 0, + 0x1D300, 0x1D356, 0, 0x1D360, 0x1D378, 0, 0x1D400, 0x1D454, 0, 0x1D456, 0x1D49C, 0, + 0x1D49E, 0x1D49F, 0, 0x1D4A2, 0x1D4A2, 0, 0x1D4A5, 0x1D4A6, 0, 0x1D4A9, 0x1D4AC, 0, + 0x1D4AE, 0x1D4B9, 0, 0x1D4BB, 0x1D4BB, 0, 0x1D4BD, 0x1D4C3, 0, 0x1D4C5, 0x1D505, 0, + 0x1D507, 0x1D50A, 0, 0x1D50D, 0x1D514, 0, 0x1D516, 0x1D51C, 0, 0x1D51E, 0x1D539, 0, + 0x1D53B, 0x1D53E, 0, 0x1D540, 0x1D544, 0, 0x1D546, 0x1D546, 0, 0x1D54A, 0x1D550, 0, + 0x1D552, 0x1D6A5, 0, 0x1D6A8, 0x1D7CB, 0, 0x1D7CE, 0x1DA8B, 0, 0x1DA9B, 0x1DA9F, 0, + 0x1DAA1, 0x1DAAF, 0, 0x1DF00, 0x1DF1E, 0, 0x1DF25, 0x1DF2A, 0, 0x1E030, 0x1E06D, 0, + 0x1E100, 0x1E12C, 0, 0x1E137, 0x1E13D, 0, 0x1E140, 0x1E149, 0, 0x1E14E, 0x1E14F, 0, + 0x1E290, 0x1E2AD, 0, 0x1E2C0, 0x1E2EB, 0, 0x1E2F0, 0x1E2F9, 0, 0x1E2FF, 0x1E2FF, 0, + 0x1E4D0, 0x1E4EB, 0, 0x1E4F0, 0x1E4F9, 0, 0x1E5D0, 0x1E5ED, 0, 0x1E5F0, 0x1E5FA, 0, + 0x1E5FF, 0x1E5FF, 0, 0x1E6C0, 0x1E6DE, 0, 0x1E6E0, 0x1E6E2, 0, 0x1E6E4, 0x1E6E5, 0, + 0x1E6E7, 0x1E6ED, 0, 0x1E6F0, 0x1E6F4, 0, 0x1E6FE, 0x1E6FF, 0, 0x1E7E0, 0x1E7E6, 0, + 0x1E7E8, 0x1E7EB, 0, 0x1E7ED, 0x1E7EE, 0, 0x1E7F0, 0x1E7FE, 0, 0x1E800, 0x1E8C4, 0, + 0x1E8C7, 0x1E8CF, 0, 0x1E900, 0x1E943, 0, 0x1E94B, 0x1E94B, 0, 0x1E950, 0x1E959, 0, + 0x1E95E, 0x1E95F, 0, 0x1EC71, 0x1ECB4, 0, 0x1ED01, 0x1ED3D, 0, 0x1EE00, 0x1EE03, 0, + 0x1EE05, 0x1EE1F, 0, 0x1EE21, 0x1EE22, 0, 0x1EE24, 0x1EE24, 0, 0x1EE27, 0x1EE27, 0, + 0x1EE29, 0x1EE32, 0, 0x1EE34, 0x1EE37, 0, 0x1EE39, 0x1EE39, 0, 0x1EE3B, 0x1EE3B, 0, + 0x1EE42, 0x1EE42, 0, 0x1EE47, 0x1EE47, 0, 0x1EE49, 0x1EE49, 0, 0x1EE4B, 0x1EE4B, 0, + 0x1EE4D, 0x1EE4F, 0, 0x1EE51, 0x1EE52, 0, 0x1EE54, 0x1EE54, 0, 0x1EE57, 0x1EE57, 0, + 0x1EE59, 0x1EE59, 0, 0x1EE5B, 0x1EE5B, 0, 0x1EE5D, 0x1EE5D, 0, 0x1EE5F, 0x1EE5F, 0, + 0x1EE61, 0x1EE62, 0, 0x1EE64, 0x1EE64, 0, 0x1EE67, 0x1EE6A, 0, 0x1EE6C, 0x1EE72, 0, + 0x1EE74, 0x1EE77, 0, 0x1EE79, 0x1EE7C, 0, 0x1EE7E, 0x1EE7E, 0, 0x1EE80, 0x1EE89, 0, + 0x1EE8B, 0x1EE9B, 0, 0x1EEA1, 0x1EEA3, 0, 0x1EEA5, 0x1EEA9, 0, 0x1EEAB, 0x1EEBB, 0, + 0x1EEF0, 0x1EEF1, 0, 0x1F000, 0x1F02B, 0, 0x1F030, 0x1F093, 0, 0x1F0A0, 0x1F0AE, 0, + 0x1F0B1, 0x1F0BF, 0, 0x1F0C1, 0x1F0CF, 0, 0x1F0D1, 0x1F0F5, 0, 0x1F100, 0x1F1AD, 0, + }; + } + + private static int[] rangeChunk2() { + return new int[] { + 0x1F1E6, 0x1F202, 0, 0x1F210, 0x1F23B, 0, 0x1F240, 0x1F248, 0, 0x1F250, 0x1F251, 0, + 0x1F260, 0x1F265, 0, 0x1F300, 0x1F6D8, 0, 0x1F6DC, 0x1F6EC, 0, 0x1F6F0, 0x1F6FC, 0, + 0x1F700, 0x1F7D9, 0, 0x1F7E0, 0x1F7EB, 0, 0x1F7F0, 0x1F7F0, 0, 0x1F800, 0x1F80B, 0, + 0x1F810, 0x1F847, 0, 0x1F850, 0x1F859, 0, 0x1F860, 0x1F887, 0, 0x1F890, 0x1F8AD, 0, + 0x1F8B0, 0x1F8BB, 0, 0x1F8C0, 0x1F8C1, 0, 0x1F8D0, 0x1F8D8, 0, 0x1F900, 0x1FA57, 0, + 0x1FA60, 0x1FA6D, 0, 0x1FA70, 0x1FA7C, 0, 0x1FA80, 0x1FA8A, 0, 0x1FA8E, 0x1FAC6, 0, + 0x1FAC8, 0x1FAC8, 0, 0x1FACD, 0x1FADC, 0, 0x1FADF, 0x1FAEA, 0, 0x1FAEF, 0x1FAF8, 0, + 0x1FB00, 0x1FB92, 0, 0x1FB94, 0x1FBFA, 0, 0x20000, 0x2A6DF, 0, 0x2A700, 0x2B81D, 0, + 0x2B820, 0x2CEAD, 0, 0x2CEB0, 0x2EBE0, 0, 0x2EBF0, 0x2EE5D, 0, 0x2F800, 0x2FA1D, 0, + 0x30000, 0x3134A, 0, 0x31350, 0x33479, 0, 0xE0001, 0xE0001, 0, 0xE0020, 0xE007F, 0, + 0xE0100, 0xE01EF, 0, 0xF0000, 0xFFFFD, 0, 0x100000, 0x10FFFD, 0, 0x334, 0x338, 1, + 0x1CD4, 0x1CD4, 1, 0x1CE2, 0x1CE8, 1, 0x20D2, 0x20D3, 1, 0x20D8, 0x20DA, 1, + 0x20E5, 0x20E6, 1, 0x20EA, 0x20EB, 1, 0x10A39, 0x10A39, 1, 0x16AF0, 0x16AF4, 1, + 0x1BC9E, 0x1BC9E, 1, 0x1D167, 0x1D169, 1, 0x16FF0, 0x16FF1, 2, 0x93C, 0x93C, 3, + 0x9BC, 0x9BC, 3, 0xA3C, 0xA3C, 3, 0xABC, 0xABC, 3, 0xB3C, 0xB3C, 3, + 0xC3C, 0xC3C, 3, 0xCBC, 0xCBC, 3, 0x1037, 0x1037, 3, 0x1B34, 0x1B34, 3, + 0x1BE6, 0x1BE6, 3, 0x1C37, 0x1C37, 3, 0xA9B3, 0xA9B3, 3, 0x110BA, 0x110BA, 3, + 0x11173, 0x11173, 3, 0x111CA, 0x111CA, 3, 0x11236, 0x11236, 3, 0x112E9, 0x112E9, 3, + 0x1133B, 0x1133C, 3, 0x11446, 0x11446, 3, 0x114C3, 0x114C3, 3, 0x115C0, 0x115C0, 3, + 0x116B7, 0x116B7, 3, 0x1183A, 0x1183A, 3, 0x11943, 0x11943, 3, 0x11D42, 0x11D42, 3, + 0x1E94A, 0x1E94A, 3, 0x3099, 0x309A, 4, 0x94D, 0x94D, 5, 0x9CD, 0x9CD, 5, + 0xA4D, 0xA4D, 5, 0xACD, 0xACD, 5, 0xB4D, 0xB4D, 5, 0xBCD, 0xBCD, 5, + 0xC4D, 0xC4D, 5, 0xCCD, 0xCCD, 5, 0xD3B, 0xD3C, 5, 0xD4D, 0xD4D, 5, + 0xDCA, 0xDCA, 5, 0xE3A, 0xE3A, 5, 0xEBA, 0xEBA, 5, 0xF84, 0xF84, 5, + 0x1039, 0x103A, 5, 0x1714, 0x1715, 5, 0x1734, 0x1734, 5, 0x17D2, 0x17D2, 5, + 0x1A60, 0x1A60, 5, 0x1B44, 0x1B44, 5, 0x1BAA, 0x1BAB, 5, 0x1BF2, 0x1BF3, 5, + 0x2D7F, 0x2D7F, 5, 0xA806, 0xA806, 5, 0xA82C, 0xA82C, 5, 0xA8C4, 0xA8C4, 5, + 0xA953, 0xA953, 5, 0xA9C0, 0xA9C0, 5, 0xAAF6, 0xAAF6, 5, 0xABED, 0xABED, 5, + 0x10A3F, 0x10A3F, 5, 0x11046, 0x11046, 5, 0x11070, 0x11070, 5, 0x1107F, 0x1107F, 5, + 0x110B9, 0x110B9, 5, 0x11133, 0x11134, 5, 0x111C0, 0x111C0, 5, 0x11235, 0x11235, 5, + 0x112EA, 0x112EA, 5, 0x1134D, 0x1134D, 5, 0x113CE, 0x113D0, 5, 0x11442, 0x11442, 5, + 0x114C2, 0x114C2, 5, 0x115BF, 0x115BF, 5, 0x1163F, 0x1163F, 5, 0x116B6, 0x116B6, 5, + 0x1172B, 0x1172B, 5, 0x11839, 0x11839, 5, 0x1193D, 0x1193E, 5, 0x119E0, 0x119E0, 5, + 0x11A34, 0x11A34, 5, 0x11A47, 0x11A47, 5, 0x11A99, 0x11A99, 5, 0x11C3F, 0x11C3F, 5, + 0x11D44, 0x11D45, 5, 0x11D97, 0x11D97, 5, 0x11F41, 0x11F42, 5, 0x1612F, 0x1612F, 5, + 0x5B0, 0x5B0, 6, 0x5B1, 0x5B1, 7, 0x5B2, 0x5B2, 8, 0x5B3, 0x5B3, 9, + 0x5B4, 0x5B4, 10, 0x5B5, 0x5B5, 11, 0x5B6, 0x5B6, 12, 0x5B7, 0x5B7, 13, + 0x5B8, 0x5B8, 14, 0x5C7, 0x5C7, 14, 0x5B9, 0x5BA, 15, 0x5BB, 0x5BB, 16, + 0x5BC, 0x5BC, 17, 0x5BD, 0x5BD, 18, 0x5BF, 0x5BF, 19, 0x5C1, 0x5C1, 20, + 0x5C2, 0x5C2, 21, 0xFB1E, 0xFB1E, 22, 0x64B, 0x64B, 23, 0x8F0, 0x8F0, 23, + 0x64C, 0x64C, 24, 0x8F1, 0x8F1, 24, 0x64D, 0x64D, 25, 0x8F2, 0x8F2, 25, + 0x618, 0x618, 26, 0x64E, 0x64E, 26, 0x619, 0x619, 27, 0x64F, 0x64F, 27, + 0x61A, 0x61A, 28, 0x650, 0x650, 28, 0x651, 0x651, 29, 0x652, 0x652, 30, + 0x670, 0x670, 31, 0x711, 0x711, 32, 0xC55, 0xC55, 33, 0xC56, 0xC56, 34, + 0xE38, 0xE39, 35, 0xE48, 0xE4B, 36, 0xEB8, 0xEB9, 37, 0xEC8, 0xECB, 38, + 0xF71, 0xF71, 39, 0xF72, 0xF72, 40, 0xF7A, 0xF7D, 40, 0xF80, 0xF80, 40, + 0xF74, 0xF74, 41, 0x321, 0x322, 42, 0x327, 0x328, 42, 0x1DD0, 0x1DD0, 42, + 0x1DCE, 0x1DCE, 43, 0x31B, 0x31B, 44, 0xF39, 0xF39, 44, 0x1D165, 0x1D166, 44, + 0x1D16E, 0x1D172, 44, 0x1DFA, 0x1DFA, 45, 0x302A, 0x302A, 45, 0x316, 0x319, 46, + 0x31C, 0x320, 46, 0x323, 0x326, 46, 0x329, 0x333, 46, 0x339, 0x33C, 46, + 0x347, 0x349, 46, 0x34D, 0x34E, 46, 0x353, 0x356, 46, 0x359, 0x35A, 46, + 0x591, 0x591, 46, 0x596, 0x596, 46, 0x59B, 0x59B, 46, 0x5A2, 0x5A7, 46, + 0x5AA, 0x5AA, 46, 0x5C5, 0x5C5, 46, 0x655, 0x656, 46, 0x65C, 0x65C, 46, + 0x65F, 0x65F, 46, 0x6E3, 0x6E3, 46, 0x6EA, 0x6EA, 46, 0x6ED, 0x6ED, 46, + 0x731, 0x731, 46, 0x734, 0x734, 46, 0x737, 0x739, 46, 0x73B, 0x73C, 46, + 0x73E, 0x73E, 46, 0x742, 0x742, 46, 0x744, 0x744, 46, 0x746, 0x746, 46, + 0x748, 0x748, 46, 0x7F2, 0x7F2, 46, 0x7FD, 0x7FD, 46, 0x859, 0x85B, 46, + 0x899, 0x89B, 46, 0x8CF, 0x8D3, 46, 0x8E3, 0x8E3, 46, 0x8E6, 0x8E6, 46, + 0x8E9, 0x8E9, 46, 0x8ED, 0x8EF, 46, 0x8F6, 0x8F6, 46, 0x8F9, 0x8FA, 46, + 0x952, 0x952, 46, 0xF18, 0xF19, 46, 0xF35, 0xF35, 46, 0xF37, 0xF37, 46, + 0xFC6, 0xFC6, 46, 0x108D, 0x108D, 46, 0x193B, 0x193B, 46, 0x1A18, 0x1A18, 46, + 0x1A7F, 0x1A7F, 46, 0x1AB5, 0x1ABA, 46, 0x1ABD, 0x1ABD, 46, 0x1ABF, 0x1AC0, 46, + 0x1AC3, 0x1AC4, 46, 0x1ACA, 0x1ACA, 46, 0x1ADD, 0x1ADD, 46, 0x1AE6, 0x1AE6, 46, + 0x1B6C, 0x1B6C, 46, 0x1CD5, 0x1CD9, 46, 0x1CDC, 0x1CDF, 46, 0x1CED, 0x1CED, 46, + 0x1DC2, 0x1DC2, 46, 0x1DCA, 0x1DCA, 46, 0x1DCF, 0x1DCF, 46, 0x1DF9, 0x1DF9, 46, + 0x1DFD, 0x1DFD, 46, 0x1DFF, 0x1DFF, 46, 0x20E8, 0x20E8, 46, 0x20EC, 0x20EF, 46, + 0xA92B, 0xA92D, 46, 0xAAB4, 0xAAB4, 46, 0xFE27, 0xFE2D, 46, 0x101FD, 0x101FD, 46, + 0x102E0, 0x102E0, 46, 0x10A0D, 0x10A0D, 46, 0x10A3A, 0x10A3A, 46, 0x10AE6, 0x10AE6, 46, + 0x10EFA, 0x10EFB, 46, 0x10EFD, 0x10EFF, 46, 0x10F46, 0x10F47, 46, 0x10F4B, 0x10F4B, 46, + 0x10F4D, 0x10F50, 46, 0x10F83, 0x10F83, 46, 0x10F85, 0x10F85, 46, 0x1D17B, 0x1D182, 46, + 0x1D18A, 0x1D18B, 46, 0x1E4EE, 0x1E4EE, 46, 0x1E5EF, 0x1E5EF, 46, 0x1E8D0, 0x1E8D6, 46, + 0x59A, 0x59A, 47, 0x5AD, 0x5AD, 47, 0x1939, 0x1939, 47, 0x302D, 0x302D, 47, + 0x302E, 0x302F, 48, 0x1D16D, 0x1D16D, 49, 0x5AE, 0x5AE, 50, 0x18A9, 0x18A9, 50, + 0x1DF7, 0x1DF8, 50, 0x302B, 0x302B, 50, 0x300, 0x314, 51, 0x33D, 0x344, 51, + 0x346, 0x346, 51, 0x34A, 0x34C, 51, 0x350, 0x352, 51, 0x357, 0x357, 51, + 0x35B, 0x35B, 51, 0x363, 0x36F, 51, 0x483, 0x487, 51, 0x592, 0x595, 51, + 0x597, 0x599, 51, 0x59C, 0x5A1, 51, 0x5A8, 0x5A9, 51, 0x5AB, 0x5AC, 51, + 0x5AF, 0x5AF, 51, 0x5C4, 0x5C4, 51, 0x610, 0x617, 51, 0x653, 0x654, 51, + 0x657, 0x65B, 51, 0x65D, 0x65E, 51, 0x6D6, 0x6DC, 51, 0x6DF, 0x6E2, 51, + 0x6E4, 0x6E4, 51, 0x6E7, 0x6E8, 51, 0x6EB, 0x6EC, 51, 0x730, 0x730, 51, + 0x732, 0x733, 51, 0x735, 0x736, 51, 0x73A, 0x73A, 51, 0x73D, 0x73D, 51, + 0x73F, 0x741, 51, 0x743, 0x743, 51, 0x745, 0x745, 51, 0x747, 0x747, 51, + 0x749, 0x74A, 51, 0x7EB, 0x7F1, 51, 0x7F3, 0x7F3, 51, 0x816, 0x819, 51, + 0x81B, 0x823, 51, 0x825, 0x827, 51, 0x829, 0x82D, 51, 0x897, 0x898, 51, + 0x89C, 0x89F, 51, 0x8CA, 0x8CE, 51, 0x8D4, 0x8E1, 51, 0x8E4, 0x8E5, 51, + 0x8E7, 0x8E8, 51, 0x8EA, 0x8EC, 51, 0x8F3, 0x8F5, 51, 0x8F7, 0x8F8, 51, + 0x8FB, 0x8FF, 51, 0x951, 0x951, 51, 0x953, 0x954, 51, 0x9FE, 0x9FE, 51, + 0xF82, 0xF83, 51, 0xF86, 0xF87, 51, 0x135D, 0x135F, 51, 0x17DD, 0x17DD, 51, + 0x193A, 0x193A, 51, 0x1A17, 0x1A17, 51, 0x1A75, 0x1A7C, 51, 0x1AB0, 0x1AB4, 51, + 0x1ABB, 0x1ABC, 51, 0x1AC1, 0x1AC2, 51, 0x1AC5, 0x1AC9, 51, 0x1ACB, 0x1ADC, 51, + 0x1AE0, 0x1AE5, 51, 0x1AE7, 0x1AEA, 51, 0x1B6B, 0x1B6B, 51, 0x1B6D, 0x1B73, 51, + 0x1CD0, 0x1CD2, 51, 0x1CDA, 0x1CDB, 51, 0x1CE0, 0x1CE0, 51, 0x1CF4, 0x1CF4, 51, + 0x1CF8, 0x1CF9, 51, 0x1DC0, 0x1DC1, 51, 0x1DC3, 0x1DC9, 51, 0x1DCB, 0x1DCC, 51, + 0x1DD1, 0x1DF5, 51, 0x1DFB, 0x1DFB, 51, 0x1DFE, 0x1DFE, 51, 0x20D0, 0x20D1, 51, + 0x20D4, 0x20D7, 51, 0x20DB, 0x20DC, 51, 0x20E1, 0x20E1, 51, 0x20E7, 0x20E7, 51, + 0x20E9, 0x20E9, 51, 0x20F0, 0x20F0, 51, 0x2CEF, 0x2CF1, 51, 0x2DE0, 0x2DFF, 51, + 0xA66F, 0xA66F, 51, 0xA674, 0xA67D, 51, 0xA69E, 0xA69F, 51, 0xA6F0, 0xA6F1, 51, + 0xA8E0, 0xA8F1, 51, 0xAAB0, 0xAAB0, 51, 0xAAB2, 0xAAB3, 51, 0xAAB7, 0xAAB8, 51, + 0xAABE, 0xAABF, 51, 0xAAC1, 0xAAC1, 51, 0xFE20, 0xFE26, 51, 0xFE2E, 0xFE2F, 51, + 0x10376, 0x1037A, 51, 0x10A0F, 0x10A0F, 51, 0x10A38, 0x10A38, 51, 0x10AE5, 0x10AE5, 51, + }; + } + + private static int[] rangeChunk3() { + return new int[] { + 0x10D24, 0x10D27, 51, 0x10D69, 0x10D6D, 51, 0x10EAB, 0x10EAC, 51, 0x10F48, 0x10F4A, 51, + 0x10F4C, 0x10F4C, 51, 0x10F82, 0x10F82, 51, 0x10F84, 0x10F84, 51, 0x11100, 0x11102, 51, + 0x11366, 0x1136C, 51, 0x11370, 0x11374, 51, 0x1145E, 0x1145E, 51, 0x16B30, 0x16B36, 51, + 0x1D185, 0x1D189, 51, 0x1D1AA, 0x1D1AD, 51, 0x1D242, 0x1D244, 51, 0x1E000, 0x1E006, 51, + 0x1E008, 0x1E018, 51, 0x1E01B, 0x1E021, 51, 0x1E023, 0x1E024, 51, 0x1E026, 0x1E02A, 51, + 0x1E08F, 0x1E08F, 51, 0x1E130, 0x1E136, 51, 0x1E2AE, 0x1E2AE, 51, 0x1E2EC, 0x1E2EF, 51, + 0x1E4EF, 0x1E4EF, 51, 0x1E5EE, 0x1E5EE, 51, 0x1E6E3, 0x1E6E3, 51, 0x1E6E6, 0x1E6E6, 51, + 0x1E6EE, 0x1E6EF, 51, 0x1E6F5, 0x1E6F5, 51, 0x1E944, 0x1E949, 51, 0x315, 0x315, 52, + 0x31A, 0x31A, 52, 0x358, 0x358, 52, 0x1DF6, 0x1DF6, 52, 0x302C, 0x302C, 52, + 0x1E4EC, 0x1E4ED, 52, 0x35C, 0x35C, 53, 0x35F, 0x35F, 53, 0x362, 0x362, 53, + 0x1DFC, 0x1DFC, 53, 0x35D, 0x35E, 54, 0x360, 0x361, 54, 0x1AEB, 0x1AEB, 54, + 0x1DCD, 0x1DCD, 54, 0x345, 0x345, 55, + }; + } + + private static final int[][] RANGE_CHUNKS = { + rangeChunk0(), rangeChunk1(), rangeChunk2(), rangeChunk3() + }; + + private static final String[] ALIASES = { + "0", "0", "1", "1", "10", "10", "103", "103", + "107", "107", "11", "11", "118", "118", "12", "12", + "122", "122", "129", "129", "13", "13", "130", "130", + "132", "132", "133", "133", "14", "14", "15", "15", + "16", "16", "17", "17", "18", "18", "19", "19", + "20", "20", "200", "200", "202", "202", "21", "21", + "214", "214", "216", "216", "218", "218", "22", "22", + "220", "220", "222", "222", "224", "224", "226", "226", + "228", "228", "23", "23", "230", "230", "232", "232", + "233", "233", "234", "234", "24", "24", "240", "240", + "25", "25", "26", "26", "27", "27", "28", "28", + "29", "29", "30", "30", "31", "31", "32", "32", + "33", "33", "34", "34", "35", "35", "36", "36", + "6", "6", "7", "7", "8", "8", "84", "84", + "9", "9", "91", "91", "a", "230", "above", "230", + "aboveleft", "228", "aboveright", "232", "al", "228", "ar", "232", + "ata", "214", "atar", "216", "atb", "202", "atbl", "200", + "attachedabove", "214", "attachedaboveright", "216", "attachedbelow", "202", "attachedbelowleft", "200", + "b", "220", "below", "220", "belowleft", "218", "belowright", "222", + "bl", "218", "br", "222", "ccc10", "10", "ccc103", "103", + "ccc107", "107", "ccc11", "11", "ccc118", "118", "ccc12", "12", + "ccc122", "122", "ccc129", "129", "ccc13", "13", "ccc130", "130", + "ccc132", "132", "ccc133", "133", "ccc14", "14", "ccc15", "15", + "ccc16", "16", "ccc17", "17", "ccc18", "18", "ccc19", "19", + "ccc20", "20", "ccc21", "21", "ccc22", "22", "ccc23", "23", + "ccc24", "24", "ccc25", "25", "ccc26", "26", "ccc27", "27", + "ccc28", "28", "ccc29", "29", "ccc30", "30", "ccc31", "31", + "ccc32", "32", "ccc33", "33", "ccc34", "34", "ccc35", "35", + "ccc36", "36", "ccc84", "84", "ccc91", "91", "da", "234", + "db", "233", "doubleabove", "234", "doublebelow", "233", "hanr", "6", + "hanreading", "6", "iotasubscript", "240", "is", "240", "kanavoicing", "8", + "kv", "8", "l", "224", "left", "224", "nk", "7", + "notreordered", "0", "nr", "0", "nukta", "7", "ov", "1", + "overlay", "1", "r", "226", "right", "226", "virama", "9", + "vr", "9", + }; + + private static final UnicodeSet[] SETS = buildSets(); + private static final Map ALIAS_INDEX = buildAliasIndex(); + + static UnicodeSet resolve(String value) { + Integer index = ALIAS_INDEX.get(normalizeValue(value)); + return index == null ? null : SETS[index]; + } + + private static UnicodeSet[] buildSets() { + UnicodeSet[] sets = new UnicodeSet[VALUES.length]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + UnicodeSet covered = new UnicodeSet(); + for (int[] ranges : RANGE_CHUNKS) { + for (int i = 0; i < ranges.length; i += 3) { + sets[ranges[i + 2]].add(ranges[i], ranges[i + 1]); + covered.add(ranges[i], ranges[i + 1]); + } + } + sets[DEFAULT_VALUE_INDEX].addAll( + new UnicodeSet(0, 0x10FFFF).removeAll(covered)); + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + + private static Map buildAliasIndex() { + Map valueIndexes = new HashMap<>(); + for (int i = 0; i < VALUES.length; i++) valueIndexes.put(VALUES[i], i); + Map indexes = new HashMap<>(); + for (int i = 0; i < ALIASES.length; i += 2) { + indexes.put(ALIASES[i], valueIndexes.get(Integer.parseInt(ALIASES[i + 1]))); + } + return Map.copyOf(indexes); + } + + private static String normalizeValue(String value) { + String normalized = value.trim(); + if (normalized.startsWith(":\\A") && normalized.endsWith("\\z:") + && normalized.length() > 6) { + normalized = normalized.substring(3, normalized.length() - 3); + } + StringBuilder loose = new StringBuilder(normalized.length()); + boolean numeric = true; + for (int i = 0; i < normalized.length(); i++) { + char ch = normalized.charAt(i); + if (Character.isWhitespace(ch) || ch == '-' || ch == '_' || ch == '+') continue; + if (!Character.isDigit(ch)) numeric = false; + loose.append(Character.toLowerCase(ch)); + } + if (!numeric) return loose.toString(); + int first = 0; + while (first + 1 < loose.length() && loose.charAt(first) == '0') first++; + return loose.substring(first); + } + + private PerlUnicodeCombiningClassData() { + } +} diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index 4823ebbf17..1ae52ce5b0 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -948,6 +948,17 @@ && isGeneralCategoryProperty(alias.substring(0, assignment))) { } return category; } + if (assignment > 0 && assignment < alias.length() - 1 + && isCanonicalCombiningClassProperty(alias.substring(0, assignment))) { + UnicodeSet combiningClass = PerlUnicodeCombiningClassData.resolve( + alias.substring(assignment + 1)); + if (combiningClass == null) { + throw new IllegalArgumentException( + "Unsupported Canonical_Combining_Class value: " + + alias.substring(assignment + 1).trim()); + } + return combiningClass; + } if (assignment > 0 && assignment < alias.length() - 1) { Boolean value = perlBooleanPropertyValue(alias.substring(assignment + 1)); if (value != null) { @@ -1014,6 +1025,13 @@ private static boolean isGeneralCategoryProperty(String property) { }; } + private static boolean isCanonicalCombiningClassProperty(String property) { + return switch (loosePropertyName(property)) { + case "ccc", "canonicalcombiningclass" -> true; + default -> false; + }; + } + private static int propertyValueDelimiter(String property) { int equals = property.indexOf('='); if (equals >= 0) return equals; @@ -1213,6 +1231,11 @@ private static String wrapProperty(String property, boolean negated) { } private static String wrapCharClass(String pattern, boolean negated) { + if (pattern.isEmpty()) { + return negated + ? "[\\x{0}-\\x{10FFFF}]" + : "[^\\x{0}-\\x{10FFFF}]"; + } return negated ? "[^" + pattern + "]" : "[" + pattern + "]"; } diff --git a/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeCombiningClassDataTest.java b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeCombiningClassDataTest.java new file mode 100644 index 0000000000..4225b3302e --- /dev/null +++ b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeCombiningClassDataTest.java @@ -0,0 +1,31 @@ +package org.perlonjava.runtime.regex; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; + +@Tag("unit") +class PerlUnicodeCombiningClassDataTest { + @Test + void usesPinnedPerl544Unicode17DataAndAliases() { + assertEquals("17.0.0", PerlUnicodeCombiningClassData.UNICODE_VERSION); + assertTrue(PerlUnicodeCombiningClassData.resolve("A").contains(0x301)); + assertTrue(PerlUnicodeCombiningClassData.resolve("+00_230").contains(0x301)); + assertTrue(PerlUnicodeCombiningClassData.resolve(":\\AAbove\\z:").contains(0x301)); + assertTrue(PerlUnicodeCombiningClassData.resolve("Overlay").contains(0x334)); + assertTrue(PerlUnicodeCombiningClassData.resolve("10").contains(0x5b0)); + assertTrue(PerlUnicodeCombiningClassData.resolve("NR").contains('A')); + assertTrue(PerlUnicodeCombiningClassData.resolve("NR").contains(0x378)); + assertFalse(PerlUnicodeCombiningClassData.resolve("230").contains('A')); + assertTrue(PerlUnicodeCombiningClassData.resolve("133").isEmpty()); + assertEquals("[^\\x{0}-\\x{10FFFF}]", + UnicodeResolver.translateUnicodeProperty("ccc=133", false)); + assertEquals("[\\x{0}-\\x{10FFFF}]", + UnicodeResolver.translateUnicodeProperty("ccc=133", true)); + assertNull(PerlUnicodeCombiningClassData.resolve("not_a_class")); + } +} diff --git a/src/test/resources/unit/regex/unicode_canonical_combining_class_pinned.t b/src/test/resources/unit/regex/unicode_canonical_combining_class_pinned.t new file mode 100644 index 0000000000..f17675b0ee --- /dev/null +++ b/src/test/resources/unit/regex/unicode_canonical_combining_class_pinned.t @@ -0,0 +1,32 @@ +use strict; +use warnings; +use Test::More tests => 24; +no warnings 'experimental::uniprop_wildcards'; + +my $above = "\x{301}"; +my $overlay = "\x{334}"; + +ok($above =~ /\p{ccc=230}/, 'numeric canonical combining class'); +ok($above =~ /\p{ccc=Above}/, 'long value alias'); +ok($above =~ /\p{ccc=A}/, 'short value alias'); +ok($above =~ /\p{Canonical_Combining_Class=Above}/, 'long property alias'); +ok($above =~ /\p{ccc: 230}/, 'colon property delimiter'); +ok($above =~ /\p{Is_Canonical_Combining_Class=Above}/, 'Is property prefix'); +ok($above =~ /\p{ccc=+00_230}/, 'loose numeric value'); +ok($above =~ /\p{ccc=:\AAbove\z:}/, 'anchored wildcard value'); +ok($overlay =~ /\p{ccc=1}/, 'overlay numeric value'); +ok($overlay =~ /\p{ccc=OV}/, 'overlay short alias'); +ok($overlay =~ /\p{ccc=Overlay}/, 'overlay long alias'); +ok('A' =~ /\p{ccc=0}/, 'not-reordered numeric value'); +ok('A' =~ /\p{ccc=NR}/, 'not-reordered short alias'); +ok("\x{378}" =~ /\p{ccc=0}/, 'unassigned code point uses missing default'); +ok("\x{378}" =~ /\p{ccc=NR}/, 'unassigned code point uses default alias'); +ok('A' !~ /\p{ccc=133}/, 'reserved empty class never matches'); +ok('A' =~ /\P{ccc=133}/, 'negated reserved empty class matches'); +ok('A' =~ /\p{^ccc=133}/, 'inner-negated reserved empty class matches'); +ok('A' !~ /\P{^ccc=133}/, 'double-negated reserved empty class never matches'); +ok($above !~ /\p{ccc=1}/, 'exact class excludes another value'); +ok($above =~ /\P{ccc=1}/, 'outer property negation'); +ok($above =~ /\p{^ccc=1}/, 'inner property negation'); +ok($above =~ /\P{^ccc=230}/, 'double property negation'); +ok("\x{5B0}" =~ /\p{ccc=10}/, 'fixed numeric CCC alias'); From 1a0ea24cf9f12656ede05dbf31236964396ff94a Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:32 +0200 Subject: [PATCH 27/60] fix(regex): parse Perl numeric escapes natively in Joni (#1032) Squashed integration of PR #1032 at c1a7a999a. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 34 ++- docs/reference/feature-matrix.md | 2 +- .../unit/regex/numeric_escape_underscores.t | 72 ++++++ third_party/joni/src/org/joni/Lexer.java | 207 +++++++++++------- .../org/joni/test/TestPerlNumericEscapes.java | 121 ++++++++++ 5 files changed, 346 insertions(+), 90 deletions(-) create mode 100644 src/test/resources/unit/regex/numeric_escape_underscores.t create mode 100644 third_party/joni/test/org/joni/test/TestPerlNumericEscapes.java diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index af08952a18..a5fc26a5ee 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,7 +224,7 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1030, stacked +The published integration stack is preserved through draft PR #1031, stacked on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the @@ -234,9 +234,10 @@ boundaries; PR #1028 adds independently validated alpha assertion aliases and native word boundaries. PR #1029 integrates corrected global zero-width `/g` progression and pinned Perl 5.44 Unicode 17.0 Age properties. PR #1030 adds binary `ASCII_Hex_Digit` values, pinned General_Category sets, and exact native -line boundaries. The new WIP adds pinned Canonical_Combining_Class sets while -the numeric-escape, Bidi_Class, Decomposition_Type, and vertical-whitespace -slices advance independently. +line boundaries. PR #1031 adds pinned Canonical_Combining_Class sets and valid +empty-property rendering. The new WIP integrates native numeric escapes while +the Bidi_Class, Decomposition_Type, and vertical-whitespace slices advance +independently. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -378,10 +379,15 @@ assigned line-boundary gap. Six regressions versus the preceding Joni result reduced to two fatal roots. Binary `ASCII_Hex_Digit=True` routing is now closed: the focused Perl boolean-value oracle passes 16/16 on system Perl, JVM, and interpreter, and `pat.t` is restored from its zero-TAP abort to the independently -tracked test-239 runtime-eval gate. PerlOnJava3 owns the remaining unbraced byte -escape `too short multibyte code string` root in `pat_advanced{,_thr}.t` and -`pat_rt_report{,_thr}.t` as part of the overlapping native Joni numeric-escape -parser slice. +tracked test-239 runtime-eval gate. Native Joni numeric parsing now treats bare +high octal escapes as UTF-8 code points and accepts underscored braced hex and +octal escapes through U+10FFFF. The focused standard-Perl oracle has 14 ordinary +passes plus four explicitly classified TODOs on both execution backends; +`pat_rt_report{,_thr}` advances from 5 executed assertions to 73/72, and +`pat_advanced.t` reaches its later independent `Titlecase` property blocker. +Strict-regex source policy and Perl code points above U+10FFFF remain explicit +frontend/representation debt, so the forced-Java underscore compatibility pass +is retained for now. ### Completed Phases @@ -515,6 +521,10 @@ parser slice. - [x] Integrated native braced-octal parsing and missing-close/empty diagnostics plus fatal unterminated braced-hex diagnostics (`55433291a`, `913e2b583`) with exact JVM/interpreter `reg_mesg.t` parity. + - [x] Integrated native bare high-octal and underscored braced hex/octal + parsing through U+10FFFF (`f849c2ef9`, integrated as `eb907a10b`). The + focused gate has 14 ordinary passes plus four classified TODOs on both + backends and restores `pat_advanced`/`pat_rt_report` startup. - [x] Integrated native Joni alpha assertion aliases `pla`, `plb`, `nla`, `nlb`, and `atomic` (`a6255fbff`, integrated as `49d7d9648`). The focused 25-case oracle passes on both execution backends and the generated alpha @@ -590,10 +600,10 @@ parser slice. ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1030. Publish the pinned CCC - slice, then integrate accepted native numeric-escape commit `f849c2ef9` in a - new WIP PR. Preserve its restored `pat_advanced`/`pat_rt_report` startup and - explicit strict-policy and above-U+10FFFF representation debt. +1. Land review-ready PR #1024 and draft PRs #1025–#1031. Finish the combined + validation for integrated numeric-escape commit `eb907a10b`, then publish + the new WIP PR. Preserve its restored `pat_advanced`/`pat_rt_report` startup + and explicit strict-policy and above-U+10FFFF representation debt. 2. Preserve the now-complete native Joni boundary corpus at 239,866/239,866: sentence chunk 05, line chunks 06–09, and word chunk 10 must remain exact on JVM and interpreter while property and parser work continues. diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index 6a85ed5d1c..20f25d9346 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -386,7 +386,7 @@ my @copy = @{$z}; # ERROR - ✅ **Overloading**: `qr` overloading is implemented. See also [overload pragma](#pragmas). - ✅ **Python-style named groups**: `(?P...)` and `(?P=name)` are parsed natively by Joni with Perl capture numbering, duplicate-name behavior, and malformed/unknown-name diagnostics. - ✅ **Alpha assertion aliases**: `(*pla:...)`, `(*plb:...)`, `(*nla:...)`, `(*nlb:...)`, and `(*atomic:...)` are parsed natively by Joni with Perl nesting, capture numbering, backtracking, and malformed-form diagnostics. -- ❌ **Underscored numeric regex escapes**: Perl spellings such as `\x{0_0_4_1}` and `\o{0_0_1_0_1}` still rely on frontend normalization rather than native Joni parsing. Valid braced octal and unconditional missing-close/empty diagnostics are native; three `use re 'strict'` non-octal cases remain deferred. +- 🟡 **Underscored numeric regex escapes**: Joni natively parses Perl spellings such as `\x{0_0_4_1}` and `\o{0_0_1_0_1}` through U+10FFFF, including literal/class forms, bare high-octal UTF-8 code points, truncation behavior, and structural diagnostics. The frontend normalization remains for forced-Java compatibility; exact `use re 'strict'` diagnostics and Perl code points above U+10FFFF through signed IV max remain source-policy/representation debt. - ✅ **Dynamically-scoped regex variables**: Provisional captures, `$^R`, `$^N`, match positions, and callback locals follow matcher paths and unwind on backtracking. - ✅ **Recursive and Dynamic Patterns**: `(?R)`, `(?0)`, and runtime `(??{ code })` execute through Joni. Dynamic expressions may return strings or `qr//` values, nested alternatives participate in outer backtracking without changing outer grouping or capture numbering, and callback and pure-pattern recursion have engine-owned depth ceilings. diff --git a/src/test/resources/unit/regex/numeric_escape_underscores.t b/src/test/resources/unit/regex/numeric_escape_underscores.t new file mode 100644 index 0000000000..06bcaa8751 --- /dev/null +++ b/src/test/resources/unit/regex/numeric_escape_underscores.t @@ -0,0 +1,72 @@ +use strict; +use warnings; +use utf8; +use Test::More; + +my $e_acute = "\x{e9}"; +ok($e_acute =~ /\xE9/, 'unbraced hex escape matches a UTF-8 scalar'); +ok($e_acute =~ /[\xE9]/, 'unbraced hex escape works in a character class'); + +my $sharp_s = "\x{df}"; +ok($sharp_s =~ /\337/, 'unbraced high octal escape matches a UTF-8 scalar'); +ok($sharp_s =~ /[\337]/, + 'unbraced high octal escape works in a character class'); + +my $smile = "\x{1f642}"; +ok($smile =~ /\x{1_F642}/, + 'underscored braced hex escape matches a supplementary scalar'); +ok($smile =~ /[\x{1_F642}]/, + 'underscored braced hex escape works in a character class'); +ok($smile =~ /\o{373_102}/, + 'underscored braced octal escape matches a supplementary scalar'); +ok($smile =~ /[\o{373_102}]/, + 'underscored braced octal escape works in a character class'); + +my $unicode_max = chr(0x10ffff); +ok($unicode_max =~ /\x{10_FFFF}/, + 'underscored hex accepts the maximum Unicode code point'); +ok($unicode_max =~ /\o{4_177_777}/, + 'underscored octal accepts the maximum Unicode code point'); +ok('A' =~ /\x{0_0000_0041}/, + 'leading zeroes do not count toward the braced hex digit limit'); + +for my $case ( + [ q!qr/\x{41/!, qr/Missing right brace on \\x\{\}/, + 'missing hex right brace is a structural error' ], + [ q!qr/\o{101/!, qr/Missing right brace on \\o\{\}/, + 'missing octal right brace is a structural error' ], + [ q!qr/\o{}/!, qr/Empty \\o\{\}/, + 'empty octal escape is a structural error' ], +) { + my ($source, $error, $name) = @$case; + my $compiled = eval $source; + ok(!defined($compiled) && $@ =~ $error, $name); +} + +TODO: { + local $TODO = q{Perl use re 'strict' numeric-escape diagnostics need source policy}; + local $SIG{__WARN__} = sub { }; + + my $hex = eval q!use re 'strict'; qr/\x{4__1}/!; + ok(!defined($hex) && $@ =~ /Non-hex character/, + 'strict mode rejects adjacent hex underscores'); + + my $octal = eval q!use re 'strict'; qr/\o{1__01}/!; + ok(!defined($octal) && $@ =~ /Non-octal character/, + 'strict mode rejects adjacent octal underscores'); +} + +TODO: { + local $TODO = 'Joni cannot represent Perl scalars above U+10FFFF'; + no warnings 'utf8'; + + my $above_unicode = chr(0x110000); + my $hex = eval q{qr/\x{11_0000}/}; + ok($hex && $above_unicode =~ $hex, + 'underscored hex preserves a scalar above Unicode'); + my $octal = eval q{qr/\o{4_200_000}/}; + ok($octal && $above_unicode =~ $octal, + 'underscored octal preserves a scalar above Unicode'); +} + +done_testing; diff --git a/third_party/joni/src/org/joni/Lexer.java b/third_party/joni/src/org/joni/Lexer.java index 8d26e879ba..6ca7d05289 100644 --- a/third_party/joni/src/org/joni/Lexer.java +++ b/third_party/joni/src/org/joni/Lexer.java @@ -575,23 +575,13 @@ private void fetchTokenInCCFor_x() { int last = p; if (peekIs('{') && syntax.opEscXBraceHex8()) { - validatePerlBracedHexClose(); - inc(); - int num = scanUnsignedHexadecimalNumber(0, 8); - if (num < 0) newValueException(ERR_TOO_BIG_WIDE_CHAR_VALUE); - if (left()) { - int c2 = peek(); - if (enc.isXDigit(c2)) newValueException(ERR_TOO_LONG_WIDE_CHAR_VALUE); - } - - if (p > last + enc.length(bytes, last, stop) && left() && peekIs('}')) { - inc(); + if (syntax.op2OptionPerl()) { + int num = scanPerlBracedCodePoint(16, 'x'); token.type = TokenType.CODE_POINT; token.base = 16; token.setCode(num); } else { - /* can't read nothing or invalid format */ - p = last; + scanOriginalBracedHexCodePoint(last); } } else if (syntax.opEscXHex2()) { int num = scanUnsignedHexadecimalNumber(0, 2); @@ -610,78 +600,150 @@ private void fetchTokenInCCFor_x() { } } - private void validatePerlBracedHexClose() { - if (!syntax.op2OptionPerl()) return; - - int cursor = p; - while (cursor < stop) { - int code = enc.mbcToCode(bytes, cursor, stop); - if (code == '}') return; - cursor += enc.length(bytes, cursor, stop); - } - newSyntaxException(PERL_MISSING_RIGHT_BRACE_ON_HEX_ESCAPE); - } - private void fetchTokenFor_oBrace() { - int afterO = p; if (!syntax.op2OptionPerl() || !left() || !peekIs('{')) { c = env.convertBackslashValue('o'); token.setC(c); return; } + int value = scanPerlBracedCodePoint(8, 'o'); + token.type = TokenType.CODE_POINT; + token.base = 8; + token.setCode(value); + } + + private void scanOriginalBracedHexCodePoint(int last) { inc(); - int cursor = p; - int close = -1; - while (cursor < stop) { - int code = enc.mbcToCode(bytes, cursor, stop); - if (code == '}') { - close = cursor; - break; - } - cursor += enc.length(bytes, cursor, stop); + int num = scanUnsignedHexadecimalNumber(0, 8); + if (num < 0) newValueException(ERR_TOO_BIG_WIDE_CHAR_VALUE); + if (left() && enc.isXDigit(peek())) newValueException(ERR_TOO_LONG_WIDE_CHAR_VALUE); + + if (p > last + enc.length(bytes, last, stop) && left() && peekIs('}')) { + inc(); + token.type = TokenType.CODE_POINT; + token.base = 16; + token.setCode(num); + } else { + p = last; } - if (close < 0) { - newSyntaxException(PERL_MISSING_RIGHT_BRACE_ON_OCTAL_ESCAPE); + } + + private int scanPerlBracedCodePoint(int radix, char escape) { + inc(); + int close = findPerlBracedEscapeClose(escape); + int cursor = p; + while (cursor < close && isPerlEscapeWhitespace(codeAt(cursor, close))) { + cursor = nextChar(cursor, close); } + int numberStart = cursor; + long value = 0; + int significantDigitCount = 0; boolean sawDigit = false; - boolean trailingWhitespace = false; - boolean invalid = false; - int value = 0; - cursor = p; + int invalid = -1; while (cursor < close) { - int code = enc.mbcToCode(bytes, cursor, close); - if (code == ' ' || code == '\t' || code == '\n' - || code == '\r' || code == '\f') { - if (sawDigit) trailingWhitespace = true; - } else if (code >= '0' && code <= '7' && !trailingWhitespace) { + int code = codeAt(cursor, close); + int digit = perlEscapeDigit(code, radix); + if (digit >= 0) { sawDigit = true; - if (value > (Integer.MAX_VALUE - (code - '0')) / 8) { - newValueException(TOO_BIG_NUMBER); + if (value != 0 || digit != 0) significantDigitCount++; + if (radix == 16 && significantDigitCount > 8) { + newValueException(ERR_TOO_LONG_WIDE_CHAR_VALUE); } - value = value * 8 + code - '0'; - } else { - invalid = true; + if (value > (Long.MAX_VALUE - digit) / radix) { + newValueException(ERR_TOO_BIG_WIDE_CHAR_VALUE); + } + value = value * radix + digit; + cursor = nextChar(cursor, close); + continue; + } + if (code == '_' && underscoreSeparatesDigits( + cursor, close, numberStart, sawDigit, radix)) { + cursor = nextChar(cursor, close); + continue; + } + if (isPerlEscapeWhitespace(code) && onlyPerlEscapeWhitespaceAfter(cursor, close)) { + cursor = close; + break; } - cursor += enc.length(bytes, cursor, close); + invalid = code; + break; } - if (invalid) { - p = afterO; - c = env.convertBackslashValue('o'); - token.setC(c); - return; - } - if (!sawDigit) { - newSyntaxException(PERL_EMPTY_OCTAL_ESCAPE); + if (!sawDigit && invalid < 0) { + if (escape == 'o') newSyntaxException(PERL_EMPTY_OCTAL_ESCAPE); + value = 0; + } else if (invalid >= 0) { + String kind = escape == 'o' ? "octal" : "hex"; + syntaxWarn("Non-" + kind + " character '" + (char)invalid + + "' terminates \\" + escape + " early. Resolved as \"\\" + + escape + "{" + formatPerlResolvedEscape(value, radix) + "}\""); } + if (value > 0x10ffff) newValueException(ERR_TOO_BIG_WIDE_CHAR_VALUE); p = close; inc(); - token.type = TokenType.CODE_POINT; - token.base = 8; - token.setCode(value); + return (int)value; + } + + private int findPerlBracedEscapeClose(char escape) { + int cursor = p; + while (cursor < stop) { + if (codeAt(cursor, stop) == '}') return cursor; + cursor = nextChar(cursor, stop); + } + newSyntaxException(escape == 'o' + ? PERL_MISSING_RIGHT_BRACE_ON_OCTAL_ESCAPE + : PERL_MISSING_RIGHT_BRACE_ON_HEX_ESCAPE); + return stop; + } + + private boolean underscoreSeparatesDigits(int cursor, int close, int numberStart, + boolean sawDigit, int radix) { + int next = nextChar(cursor, close); + if (next >= close || perlEscapeDigit(codeAt(next, close), radix) < 0) return false; + return sawDigit || cursor == numberStart; + } + + private static int perlEscapeDigit(int code, int radix) { + int digit; + if (code >= '0' && code <= '9') { + digit = code - '0'; + } else if (code >= 'a' && code <= 'f') { + digit = code - 'a' + 10; + } else if (code >= 'A' && code <= 'F') { + digit = code - 'A' + 10; + } else { + return -1; + } + return digit < radix ? digit : -1; + } + + private static String formatPerlResolvedEscape(long value, int radix) { + String digits = Long.toString(value, radix); + int minimumWidth = radix == 8 ? 3 : 2; + return "0".repeat(Math.max(0, minimumWidth - digits.length())) + digits; + } + + private boolean onlyPerlEscapeWhitespaceAfter(int cursor, int close) { + while (cursor < close) { + if (!isPerlEscapeWhitespace(codeAt(cursor, close))) return false; + cursor = nextChar(cursor, close); + } + return true; + } + + private static boolean isPerlEscapeWhitespace(int code) { + return code == ' ' || code == '\t' || code == '\n' || code == '\r' || code == '\f'; + } + + private int codeAt(int cursor, int end) { + return enc.mbcToCode(bytes, cursor, end); + } + + private int nextChar(int cursor, int end) { + return cursor + enc.length(bytes, cursor, end); } private void fetchTokenInCCFor_u() { @@ -710,7 +772,7 @@ private void fetchTokenInCCFor_digit() { if (p == last) { /* can't read nothing. */ num = 0; /* but, it's not error */ } - if (num > 0xff && syntax.op2OptionPerl()) { + if (syntax.op2OptionPerl() && (!enc.isSingleByte() || num > 0xff)) { token.type = TokenType.CODE_POINT; token.setCode(num); } else { @@ -881,21 +943,12 @@ private void fetchTokenFor_xBrace() { int last = p; if (peekIs('{') && syntax.opEscXBraceHex8()) { - validatePerlBracedHexClose(); - inc(); - int num = scanUnsignedHexadecimalNumber(0, 8); - if (num < 0) newValueException(ERR_TOO_BIG_WIDE_CHAR_VALUE); - if (left()) { - if (enc.isXDigit(peek())) newValueException(ERR_TOO_LONG_WIDE_CHAR_VALUE); - } - - if (p > last + enc.length(bytes, last, stop) && left() && peekIs('}')) { - inc(); + if (syntax.op2OptionPerl()) { + int num = scanPerlBracedCodePoint(16, 'x'); token.type = TokenType.CODE_POINT; token.setCode(num); } else { - /* can't read nothing or invalid format */ - p = last; + scanOriginalBracedHexCodePoint(last); } } else if (syntax.opEscXHex2()) { int num = scanUnsignedHexadecimalNumber(0, 2); @@ -966,7 +1019,7 @@ private void fetchTokenFor_zero() { if (p == last) { /* can't read nothing. */ num = 0; /* but, it's not error */ } - if (num > 0xff && syntax.op2OptionPerl()) { + if (syntax.op2OptionPerl() && (!enc.isSingleByte() || num > 0xff)) { token.type = TokenType.CODE_POINT; token.setCode(num); } else { diff --git a/third_party/joni/test/org/joni/test/TestPerlNumericEscapes.java b/third_party/joni/test/org/joni/test/TestPerlNumericEscapes.java new file mode 100644 index 0000000000..96a4fcd485 --- /dev/null +++ b/third_party/joni/test/org/joni/test/TestPerlNumericEscapes.java @@ -0,0 +1,121 @@ +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni.test; + +import static org.joni.exception.ErrorMessages.ERR_TOO_BIG_WIDE_CHAR_VALUE; +import static org.joni.exception.ErrorMessages.PERL_EMPTY_OCTAL_ESCAPE; +import static org.joni.exception.ErrorMessages.PERL_MISSING_RIGHT_BRACE_ON_HEX_ESCAPE; +import static org.joni.exception.ErrorMessages.PERL_MISSING_RIGHT_BRACE_ON_OCTAL_ESCAPE; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.List; + +import org.jcodings.Encoding; +import org.jcodings.specific.ISO8859_1Encoding; +import org.jcodings.specific.UTF8Encoding; +import org.joni.Option; +import org.joni.Regex; +import org.joni.Syntax; +import org.joni.WarnCallback; +import org.joni.exception.JOniException; +import org.junit.Test; + +public class TestPerlNumericEscapes { + private static Regex compile(String pattern, Encoding encoding, WarnCallback warnings) { + byte[] bytes = pattern.getBytes(encoding == ISO8859_1Encoding.INSTANCE + ? StandardCharsets.ISO_8859_1 : StandardCharsets.UTF_8); + return new Regex(bytes, 0, bytes.length, Option.NONE, encoding, Syntax.PerlNG, warnings); + } + + private static int search(String pattern, String input) { + byte[] bytes = input.getBytes(StandardCharsets.UTF_8); + return compile(pattern, UTF8Encoding.INSTANCE, WarnCallback.NONE) + .matcher(bytes).search(0, bytes.length, Option.NONE); + } + + private static void assertSyntaxError(String pattern, String expected) { + try { + compile(pattern, UTF8Encoding.INSTANCE, WarnCallback.NONE); + fail("expected syntax error for " + pattern); + } catch (JOniException error) { + assertEquals(expected, error.getMessage()); + } + } + + @Test + public void parsesUnbracedHighCodePointsWithoutTreatingThemAsUtf8Bytes() { + assertEquals(0, search("\\x8e", "\u008e")); + assertEquals(0, search("[\\x8e]", "\u008e")); + assertEquals(0, search("\\337", "\u00df")); + assertEquals(0, search("[\\337]", "\u00df")); + assertEquals(0, search("\\x8e\\337", "\u008e\u00df")); + } + + @Test + public void preservesSingleByteOctalPatterns() { + byte[] input = {(byte)0xdf}; + Regex regex = compile("\\337", ISO8859_1Encoding.INSTANCE, WarnCallback.NONE); + assertEquals(0, regex.matcher(input).search(0, input.length, Option.NONE)); + regex = compile("[\\337]", ISO8859_1Encoding.INSTANCE, WarnCallback.NONE); + assertEquals(0, regex.matcher(input).search(0, input.length, Option.NONE)); + } + + @Test + public void acceptsPerlUnderscoresInBracedHexAndOctalEscapes() { + assertEquals(0, search("\\x{0_0_4_1}", "A")); + assertEquals(0, search("[\\x{_4_1}]", "A")); + assertEquals(0, search("\\o{0_0_1_0_1}", "A")); + assertEquals(0, search("[\\o{_1_01}]", "A")); + assertEquals(0, search("\\x{1_0_F}", "\u010f")); + assertEquals(0, search("\\o{4_1_7}", "\u010f")); + assertEquals(0, search("\\x{0_0000_0041}", "A")); + } + + @Test + public void truncatesInvalidSuffixesAndReportsWarnings() { + List warnings = new ArrayList<>(); + Regex regex = compile("\\x{41_}", UTF8Encoding.INSTANCE, warnings::add); + byte[] input = "A".getBytes(StandardCharsets.UTF_8); + assertEquals(0, regex.matcher(input).search(0, input.length, Option.NONE)); + assertTrue(warnings.get(0).contains( + "Non-hex character '_' terminates \\x early. Resolved as \"\\x{41}\"")); + + warnings.clear(); + regex = compile("\\o{1__01}", UTF8Encoding.INSTANCE, warnings::add); + input = new byte[] {1}; + assertEquals(0, regex.matcher(input).search(0, input.length, Option.NONE)); + assertTrue(warnings.get(0).contains( + "Non-octal character '_' terminates \\o early. Resolved as \"\\o{001}\"")); + } + + @Test + public void preservesLexerOwnedStructuralErrorsAndUnicodeBound() { + assertEquals(0, search("\\x{}", "\0")); + assertSyntaxError("\\o{}", PERL_EMPTY_OCTAL_ESCAPE); + assertSyntaxError("\\x{4_1", PERL_MISSING_RIGHT_BRACE_ON_HEX_ESCAPE); + assertSyntaxError("\\o{1_01", PERL_MISSING_RIGHT_BRACE_ON_OCTAL_ESCAPE); + assertSyntaxError("\\x{11_0000}", ERR_TOO_BIG_WIDE_CHAR_VALUE); + assertSyntaxError("\\o{4_200_000}", ERR_TOO_BIG_WIDE_CHAR_VALUE); + } +} From ebee1e6f595dceb9d87e3b25716d2c38078ce091 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:33 +0200 Subject: [PATCH 28/60] feat(regex): pin Perl Unicode Bidi_Class data (#1033) Squashed integration of PR #1033 at dd5bcd61b. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 32 +- .../generate_perl_unicode_bidi_class_data.pl | 266 +++++++++++ docs/reference/feature-matrix.md | 4 +- .../regex/PerlUnicodeBidiClassData.java | 428 ++++++++++++++++++ .../runtime/regex/UnicodeResolver.java | 12 + .../regex/PerlUnicodeBidiClassDataTest.java | 106 +++++ .../unit/regex/unicode_bidi_class_pinned.t | 67 +++ 7 files changed, 903 insertions(+), 12 deletions(-) create mode 100755 dev/tools/generate_perl_unicode_bidi_class_data.pl create mode 100644 src/main/java/org/perlonjava/runtime/regex/PerlUnicodeBidiClassData.java create mode 100644 src/test/java/org/perlonjava/runtime/regex/PerlUnicodeBidiClassDataTest.java create mode 100644 src/test/resources/unit/regex/unicode_bidi_class_pinned.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index a5fc26a5ee..95ae8c158c 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,7 +224,7 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1031, stacked +The published integration stack is preserved through draft PR #1032, stacked on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the @@ -235,9 +235,9 @@ native word boundaries. PR #1029 integrates corrected global zero-width `/g` progression and pinned Perl 5.44 Unicode 17.0 Age properties. PR #1030 adds binary `ASCII_Hex_Digit` values, pinned General_Category sets, and exact native line boundaries. PR #1031 adds pinned Canonical_Combining_Class sets and valid -empty-property rendering. The new WIP integrates native numeric escapes while -the Bidi_Class, Decomposition_Type, and vertical-whitespace slices advance -independently. +empty-property rendering. PR #1032 integrates native numeric escapes through +U+10FFFF. The new WIP adds pinned Bidi_Class sets while Decomposition_Type, +East_Asian_Width, and vertical-whitespace slices advance independently. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -312,6 +312,14 @@ assertions and 10 already-native line preamble assertions improve over the 31,311 baseline with zero numbered regressions. Current generated evidence is 341,280/407,367. +`Bidi_Class`/`bc` assignments now resolve all 23 values from a complete pinned +Unicode 17 partition. Ordered missing defaults, short/long and loose aliases, +directional controls, noncharacters, and unknown-value rejection are covered. +The focused oracle passes 99/99 on system Perl, JVM, and interpreter; the +combined Unicode/property/boundary smoke is 345/345 per backend. Chunk 01 gains +736 assertions to 34,252/41,843 identically on both execution backends with no +numbered regression, raising current generated evidence to 342,016/407,367. + Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while ordinary and substitution callbacks execute, without misclassifying escaped or @@ -558,6 +566,10 @@ is retained for now. aliases, ordered missing defaults, reserved empty values, and valid empty/full rendering. The focused oracle passes 24/24 and chunk 01 reaches 33,516/41,843 on both backends with zero numbered regressions. + - [x] Generated and integrated a complete pinned Unicode 17.0 Bidi_Class + partition with loose aliases and ordered missing defaults. The focused + oracle passes 99/99 and chunk 01 gains 736 assertions to 34,252/41,843 on + both backends with zero numbered regressions. - [ ] Close the remaining property failures before marking Phase 3 complete. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) @@ -600,15 +612,15 @@ is retained for now. ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1031. Finish the combined - validation for integrated numeric-escape commit `eb907a10b`, then publish - the new WIP PR. Preserve its restored `pat_advanced`/`pat_rt_report` startup - and explicit strict-policy and above-U+10FFFF representation debt. +1. Land review-ready PR #1024 and draft PRs #1025–#1032. Publish the validated + Bidi_Class slice, then integrate native vertical-whitespace commit + `1eff1db97` and the completed Decomposition_Type data handoff in separate + focused WIP PRs. 2. Preserve the now-complete native Joni boundary corpus at 239,866/239,866: sentence chunk 05, line chunks 06–09, and word chunk 10 must remain exact on JVM and interpreter while property and parser work continues. -3. Integrate the independently generated Bidi_Class and Decomposition_Type - slices, then continue the measured residual order: East_Asian_Width, binary +3. Integrate the independently generated Decomposition_Type and + East_Asian_Width slices, then continue the measured residual order: binary property values, Block, Script/Script_Extensions, Numeric_Value, Joining_Group, and break-property values. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. diff --git a/dev/tools/generate_perl_unicode_bidi_class_data.pl b/dev/tools/generate_perl_unicode_bidi_class_data.pl new file mode 100755 index 0000000000..cc4bf5d6be --- /dev/null +++ b/dev/tools/generate_perl_unicode_bidi_class_data.pl @@ -0,0 +1,266 @@ +#!/usr/bin/env perl +use strict; +use warnings; +use Digest::SHA qw(sha256_hex); +use File::Spec; +use FindBin; + +my $expected_version = '17.0.0'; +my $root = File::Spec->catdir($FindBin::Bin, '..', '..'); +my $unicore = File::Spec->catdir($root, 'perl5', 'lib', 'unicore'); +my @sources = ( + { + name => 'DerivedBidiClass-17.0.0.txt', + path => File::Spec->catfile($unicore, 'extracted', 'DBidiClass.txt'), + hash => '4867b4b7f0731ed1bfcd34cc6251211ff1542541fce0734b6fbda139ee80b3a4', + version => qr/^# DerivedBidiClass-\Q$expected_version\E\.txt$/m, + }, + { + name => 'PropertyValueAliases-17.0.0.txt', + path => File::Spec->catfile($unicore, 'PropValueAliases.txt'), + hash => '670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01', + version => qr/^# PropertyValueAliases-\Q$expected_version\E\.txt$/m, + }, + { + name => 'PropertyAliases-17.0.0.txt', + path => File::Spec->catfile($unicore, 'PropertyAliases.txt'), + hash => '4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb', + version => qr/^# PropertyAliases-\Q$expected_version\E\.txt$/m, + }, +); + +sub read_source { + my ($source) = @_; + open my $input, '<:raw', $source->{path} + or die "Cannot read $source->{path}: $!\n"; + local $/; + my $text = <$input>; + close $input or die "Cannot close $source->{path}: $!\n"; + my $actual_hash = sha256_hex($text); + die "$source->{path} SHA-256 mismatch: expected $source->{hash}, found $actual_hash\n" + unless $actual_hash eq $source->{hash}; + die "$source->{path} is not pinned Unicode $expected_version data\n" + unless $text =~ $source->{version}; + $source->{text} = $text; +} + +sub trim { + my ($text) = @_; + $text =~ s/^\s+|\s+$//g; + return $text; +} + +sub loose_name { + my ($name) = @_; + $name = lc $name; + $name =~ s/[\s_-]+//g; + return $name; +} + +sub parse_range { + my ($text) = @_; + my ($start, $end) = split /\.\./, $text; + return (hex($start), hex(defined $end ? $end : $start)); +} + +read_source($_) for @sources; + +my $version_path = File::Spec->catfile($unicore, 'version'); +open my $version_input, '<', $version_path or die "Cannot read $version_path: $!\n"; +chomp(my $unicode_version = <$version_input>); +close $version_input or die "Cannot close $version_path: $!\n"; +die "Expected Unicode $expected_version, found $unicode_version\n" + unless $unicode_version eq $expected_version; + +my (@short_values, @long_values, %alias_index); +for my $line (split /\n/, $sources[1]{text}) { + next if $line =~ /^\s*#/; + $line =~ s/#.*$//; + my @fields = map { trim($_) } split /;/, $line; + next unless @fields >= 3 && $fields[0] eq 'bc'; + my $index = scalar @short_values; + push @short_values, $fields[1]; + push @long_values, $fields[2]; + for my $alias (@fields[1 .. $#fields]) { + next unless length $alias; + my $loose = loose_name($alias); + die "Bidi_Class alias collision for '$alias'\n" + if exists $alias_index{$loose} && $alias_index{$loose} != $index; + $alias_index{$loose} = $index; + } +} +die "Expected 23 Bidi_Class values, found " . scalar(@short_values) . "\n" + unless @short_values == 23; + +my %property_aliases; +for my $line (split /\n/, $sources[2]{text}) { + next if $line =~ /^\s*#/; + $line =~ s/#.*$//; + my @fields = map { trim($_) } split /;/, $line; + next unless @fields >= 2 && ($fields[0] eq 'bc' || $fields[1] eq 'Bidi_Class'); + $property_aliases{loose_name($_)} = 1 for grep { length } @fields; +} +die "Pinned property aliases do not define bc and Bidi_Class\n" + unless $property_aliases{bc} && $property_aliases{bidiclass}; + +my (@missing, @explicit); +for my $line (split /\n/, $sources[0]{text}) { + if ($line =~ /^#\s*\@missing:\s*([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z_]+)/) { + my ($range, $value) = ($1, $2); + my ($start, $end) = parse_range($range); + my $index = $alias_index{loose_name($value)}; + die "Unknown \@missing Bidi_Class value '$value'\n" unless defined $index; + push @missing, [$start, $end, $index]; + next; + } + next unless $line =~ /^([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z_]+)/; + my ($range, $value) = ($1, $2); + my ($start, $end) = parse_range($range); + my $index = $alias_index{loose_name($value)}; + die "Unknown explicit Bidi_Class value '$value'\n" unless defined $index; + push @explicit, [$start, $end, $index]; +} +die "Bidi_Class data has no ranges or defaults\n" unless @explicit && @missing; + +my $default_index = $alias_index{loose_name('Left_To_Right')}; +my @code_value = ($default_index) x 0x110000; +for my $range (@missing) { + my ($code, $end, $index) = @$range; + $code_value[$code++] = $index while $code <= $end; +} +for my $range (@explicit) { + my ($code, $end, $index) = @$range; + $code_value[$code++] = $index while $code <= $end; +} + +my @ranges; +my ($range_start, $range_value) = (0, $code_value[0]); +for my $code (1 .. 0x10ffff) { + next if $code_value[$code] == $range_value; + push @ranges, [$range_start, $code - 1, $range_value]; + ($range_start, $range_value) = ($code, $code_value[$code]); +} +push @ranges, [$range_start, 0x10ffff, $range_value]; + +print <<'HEADER'; +/* + * Generated from Perl 5.44's pinned Unicode Character Database. Do not edit manually. + * +HEADER +for my $source (@sources) { + print " * Source: $source->{name}\n"; + for my $line (split /\n/, $source->{text}) { + next unless $line =~ /^# (?:©|Unicode and|the U\.S\.|For terms of use and license)/; + $line =~ s/^# / * /; + print "$line\n"; + } + print " *\n"; +} +print <<'HEADER_END'; + */ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; + +final class PerlUnicodeBidiClassData { +HEADER_END + +print " static final String UNICODE_VERSION = \"$unicode_version\";\n"; +print " static final String DBIDI_CLASS_SHA256 = \"$sources[0]{hash}\";\n"; +print " static final String PROP_VALUE_ALIASES_SHA256 = \"$sources[1]{hash}\";\n"; +print " static final String PROPERTY_ALIASES_SHA256 = \"$sources[2]{hash}\";\n\n"; + +print " private static final String[] SHORT_VALUES = {\n "; +print join(', ', map { qq{"$_"} } @short_values); +print "\n };\n"; +print " private static final String[] LONG_VALUES = {\n "; +print join(', ', map { qq{"$_"} } @long_values); +print "\n };\n"; + +my @aliases = sort keys %alias_index; +print " private static final String[] VALUE_ALIASES = {\n "; +print join(', ', map { qq{"$_"} } @aliases); +print "\n };\n"; +print " private static final byte[] VALUE_ALIAS_INDEX = {\n "; +print join(', ', map { $alias_index{$_} } @aliases); +print "\n };\n"; + +my @property_aliases = sort keys %property_aliases; +print " private static final String[] PROPERTY_ALIASES = {\n "; +print join(', ', map { qq{"$_"} } @property_aliases); +print "\n };\n\n"; + +print " private static final int[] RANGES = {\n"; +for (my $i = 0; $i < @ranges; $i += 4) { + my $end = $i + 3 < $#ranges ? $i + 3 : $#ranges; + print " ", join(', ', map { + sprintf '0x%X, 0x%X, %d', @{$ranges[$_]}[0, 1, 2] + } $i .. $end), ",\n"; +} +print <<'FOOTER'; + }; + + private static final UnicodeSet[] VALUE_SETS = buildValueSets(); + + static boolean isPropertyAlias(String alias) { + String loose = looseName(alias); + if (loose == null) return false; + for (String candidate : PROPERTY_ALIASES) { + if (candidate.equals(loose)) return true; + } + return false; + } + + static UnicodeSet valueSet(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : VALUE_SETS[index]; + } + + static String shortValue(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : SHORT_VALUES[index]; + } + + static String canonicalValue(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : LONG_VALUES[index]; + } + + static String[] canonicalValues() { + return LONG_VALUES.clone(); + } + + private static int valueIndex(String alias) { + String loose = looseName(alias); + if (loose == null) return -1; + for (int i = 0; i < VALUE_ALIASES.length; i++) { + if (VALUE_ALIASES[i].equals(loose)) return VALUE_ALIAS_INDEX[i]; + } + return -1; + } + + private static String looseName(String name) { + if (name == null) return null; + StringBuilder loose = new StringBuilder(name.length()); + for (int i = 0; i < name.length(); i++) { + char character = name.charAt(i); + if (character == '_' || character == '-' || Character.isWhitespace(character)) continue; + loose.append(Character.toLowerCase(character)); + } + return loose.toString(); + } + + private static UnicodeSet[] buildValueSets() { + UnicodeSet[] sets = new UnicodeSet[LONG_VALUES.length]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + for (int i = 0; i < RANGES.length; i += 3) { + sets[RANGES[i + 2]].add(RANGES[i], RANGES[i + 1]); + } + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + + private PerlUnicodeBidiClassData() { + } +} +FOOTER diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index 20f25d9346..f900c6d84b 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -378,7 +378,7 @@ my @copy = @{$z}; # ERROR - ✅ **Backreferences to Named Groups**: Using `\k` or `\g{name}` for backreferences to named groups is supported. - ✅ **Relative Backreferences**: Using `\g{-n}` for relative backreferences. - ✅ **Basic Unicode Properties**: Common `\p{...}` and `\P{...}` forms such as `\p{L}` execute through Joni. -- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, and Canonical_Combining_Class assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, and reserved empty values; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. +- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, Canonical_Combining_Class, and Bidi_Class assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, and reserved empty values; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. - ✅ **Possessive Quantifiers**: Quantifiers like `*+`, `++`, `?+`, and `{n,m}+`, which disable backtracking, are supported. - ✅ **Atomic Grouping**: Use of `(?>...)` for atomic groups is supported. - ✅ **`\K` assertion**: Keep left — in `s///`, text before `\K` is preserved; match variables reflect only the portion after `\K`. @@ -397,7 +397,7 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 341,280/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 342,016/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. - ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. - ✅ **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter. - ✅ **Unicode Word Boundaries**: Native `\b{wb}`/`\B{wb}` implement WB1–WB16 and WB999 from reproducible Perl 5.44 Unicode 17.0 word-break and extended-pictographic tables. Authoritative chunk 10 passes 19,510/19,510 identically on JVM and interpreter. diff --git a/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeBidiClassData.java b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeBidiClassData.java new file mode 100644 index 0000000000..c2fc66b28a --- /dev/null +++ b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeBidiClassData.java @@ -0,0 +1,428 @@ +/* + * Generated from Perl 5.44's pinned Unicode Character Database. Do not edit manually. + * + * Source: DerivedBidiClass-17.0.0.txt + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + * + * Source: PropertyValueAliases-17.0.0.txt + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + * + * Source: PropertyAliases-17.0.0.txt + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + * + */ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; + +final class PerlUnicodeBidiClassData { + static final String UNICODE_VERSION = "17.0.0"; + static final String DBIDI_CLASS_SHA256 = "4867b4b7f0731ed1bfcd34cc6251211ff1542541fce0734b6fbda139ee80b3a4"; + static final String PROP_VALUE_ALIASES_SHA256 = "670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01"; + static final String PROPERTY_ALIASES_SHA256 = "4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb"; + + private static final String[] SHORT_VALUES = { + "AL", "AN", "B", "BN", "CS", "EN", "ES", "ET", "FSI", "L", "LRE", "LRI", "LRO", "NSM", "ON", "PDF", "PDI", "R", "RLE", "RLI", "RLO", "S", "WS" + }; + private static final String[] LONG_VALUES = { + "Arabic_Letter", "Arabic_Number", "Paragraph_Separator", "Boundary_Neutral", "Common_Separator", "European_Number", "European_Separator", "European_Terminator", "First_Strong_Isolate", "Left_To_Right", "Left_To_Right_Embedding", "Left_To_Right_Isolate", "Left_To_Right_Override", "Nonspacing_Mark", "Other_Neutral", "Pop_Directional_Format", "Pop_Directional_Isolate", "Right_To_Left", "Right_To_Left_Embedding", "Right_To_Left_Isolate", "Right_To_Left_Override", "Segment_Separator", "White_Space" + }; + private static final String[] VALUE_ALIASES = { + "al", "an", "arabicletter", "arabicnumber", "b", "bn", "boundaryneutral", "commonseparator", "cs", "en", "es", "et", "europeannumber", "europeanseparator", "europeanterminator", "firststrongisolate", "fsi", "l", "lefttoright", "lefttorightembedding", "lefttorightisolate", "lefttorightoverride", "lre", "lri", "lro", "nonspacingmark", "nsm", "on", "otherneutral", "paragraphseparator", "pdf", "pdi", "popdirectionalformat", "popdirectionalisolate", "r", "righttoleft", "righttoleftembedding", "righttoleftisolate", "righttoleftoverride", "rle", "rli", "rlo", "s", "segmentseparator", "whitespace", "ws" + }; + private static final byte[] VALUE_ALIAS_INDEX = { + 0, 1, 0, 1, 2, 3, 3, 4, 4, 5, 6, 7, 5, 6, 7, 8, 8, 9, 9, 10, 11, 12, 10, 11, 12, 13, 13, 14, 14, 2, 15, 16, 15, 16, 17, 17, 18, 19, 20, 18, 19, 20, 21, 21, 22, 22 + }; + private static final String[] PROPERTY_ALIASES = { + "bc", "bidiclass" + }; + + private static final int[] RANGES = { + 0x0, 0x8, 3, 0x9, 0x9, 21, 0xA, 0xA, 2, 0xB, 0xB, 21, + 0xC, 0xC, 22, 0xD, 0xD, 2, 0xE, 0x1B, 3, 0x1C, 0x1E, 2, + 0x1F, 0x1F, 21, 0x20, 0x20, 22, 0x21, 0x22, 14, 0x23, 0x25, 7, + 0x26, 0x2A, 14, 0x2B, 0x2B, 6, 0x2C, 0x2C, 4, 0x2D, 0x2D, 6, + 0x2E, 0x2F, 4, 0x30, 0x39, 5, 0x3A, 0x3A, 4, 0x3B, 0x40, 14, + 0x41, 0x5A, 9, 0x5B, 0x60, 14, 0x61, 0x7A, 9, 0x7B, 0x7E, 14, + 0x7F, 0x84, 3, 0x85, 0x85, 2, 0x86, 0x9F, 3, 0xA0, 0xA0, 4, + 0xA1, 0xA1, 14, 0xA2, 0xA5, 7, 0xA6, 0xA9, 14, 0xAA, 0xAA, 9, + 0xAB, 0xAC, 14, 0xAD, 0xAD, 3, 0xAE, 0xAF, 14, 0xB0, 0xB1, 7, + 0xB2, 0xB3, 5, 0xB4, 0xB4, 14, 0xB5, 0xB5, 9, 0xB6, 0xB8, 14, + 0xB9, 0xB9, 5, 0xBA, 0xBA, 9, 0xBB, 0xBF, 14, 0xC0, 0xD6, 9, + 0xD7, 0xD7, 14, 0xD8, 0xF6, 9, 0xF7, 0xF7, 14, 0xF8, 0x2B8, 9, + 0x2B9, 0x2BA, 14, 0x2BB, 0x2C1, 9, 0x2C2, 0x2CF, 14, 0x2D0, 0x2D1, 9, + 0x2D2, 0x2DF, 14, 0x2E0, 0x2E4, 9, 0x2E5, 0x2ED, 14, 0x2EE, 0x2EE, 9, + 0x2EF, 0x2FF, 14, 0x300, 0x36F, 13, 0x370, 0x373, 9, 0x374, 0x375, 14, + 0x376, 0x37D, 9, 0x37E, 0x37E, 14, 0x37F, 0x383, 9, 0x384, 0x385, 14, + 0x386, 0x386, 9, 0x387, 0x387, 14, 0x388, 0x3F5, 9, 0x3F6, 0x3F6, 14, + 0x3F7, 0x482, 9, 0x483, 0x489, 13, 0x48A, 0x589, 9, 0x58A, 0x58A, 14, + 0x58B, 0x58C, 9, 0x58D, 0x58E, 14, 0x58F, 0x58F, 7, 0x590, 0x590, 17, + 0x591, 0x5BD, 13, 0x5BE, 0x5BE, 17, 0x5BF, 0x5BF, 13, 0x5C0, 0x5C0, 17, + 0x5C1, 0x5C2, 13, 0x5C3, 0x5C3, 17, 0x5C4, 0x5C5, 13, 0x5C6, 0x5C6, 17, + 0x5C7, 0x5C7, 13, 0x5C8, 0x5FF, 17, 0x600, 0x605, 1, 0x606, 0x607, 14, + 0x608, 0x608, 0, 0x609, 0x60A, 7, 0x60B, 0x60B, 0, 0x60C, 0x60C, 4, + 0x60D, 0x60D, 0, 0x60E, 0x60F, 14, 0x610, 0x61A, 13, 0x61B, 0x64A, 0, + 0x64B, 0x65F, 13, 0x660, 0x669, 1, 0x66A, 0x66A, 7, 0x66B, 0x66C, 1, + 0x66D, 0x66F, 0, 0x670, 0x670, 13, 0x671, 0x6D5, 0, 0x6D6, 0x6DC, 13, + 0x6DD, 0x6DD, 1, 0x6DE, 0x6DE, 14, 0x6DF, 0x6E4, 13, 0x6E5, 0x6E6, 0, + 0x6E7, 0x6E8, 13, 0x6E9, 0x6E9, 14, 0x6EA, 0x6ED, 13, 0x6EE, 0x6EF, 0, + 0x6F0, 0x6F9, 5, 0x6FA, 0x710, 0, 0x711, 0x711, 13, 0x712, 0x72F, 0, + 0x730, 0x74A, 13, 0x74B, 0x7A5, 0, 0x7A6, 0x7B0, 13, 0x7B1, 0x7BF, 0, + 0x7C0, 0x7EA, 17, 0x7EB, 0x7F3, 13, 0x7F4, 0x7F5, 17, 0x7F6, 0x7F9, 14, + 0x7FA, 0x7FC, 17, 0x7FD, 0x7FD, 13, 0x7FE, 0x815, 17, 0x816, 0x819, 13, + 0x81A, 0x81A, 17, 0x81B, 0x823, 13, 0x824, 0x824, 17, 0x825, 0x827, 13, + 0x828, 0x828, 17, 0x829, 0x82D, 13, 0x82E, 0x858, 17, 0x859, 0x85B, 13, + 0x85C, 0x85F, 17, 0x860, 0x88F, 0, 0x890, 0x891, 1, 0x892, 0x896, 0, + 0x897, 0x89F, 13, 0x8A0, 0x8C9, 0, 0x8CA, 0x8E1, 13, 0x8E2, 0x8E2, 1, + 0x8E3, 0x902, 13, 0x903, 0x939, 9, 0x93A, 0x93A, 13, 0x93B, 0x93B, 9, + 0x93C, 0x93C, 13, 0x93D, 0x940, 9, 0x941, 0x948, 13, 0x949, 0x94C, 9, + 0x94D, 0x94D, 13, 0x94E, 0x950, 9, 0x951, 0x957, 13, 0x958, 0x961, 9, + 0x962, 0x963, 13, 0x964, 0x980, 9, 0x981, 0x981, 13, 0x982, 0x9BB, 9, + 0x9BC, 0x9BC, 13, 0x9BD, 0x9C0, 9, 0x9C1, 0x9C4, 13, 0x9C5, 0x9CC, 9, + 0x9CD, 0x9CD, 13, 0x9CE, 0x9E1, 9, 0x9E2, 0x9E3, 13, 0x9E4, 0x9F1, 9, + 0x9F2, 0x9F3, 7, 0x9F4, 0x9FA, 9, 0x9FB, 0x9FB, 7, 0x9FC, 0x9FD, 9, + 0x9FE, 0x9FE, 13, 0x9FF, 0xA00, 9, 0xA01, 0xA02, 13, 0xA03, 0xA3B, 9, + 0xA3C, 0xA3C, 13, 0xA3D, 0xA40, 9, 0xA41, 0xA42, 13, 0xA43, 0xA46, 9, + 0xA47, 0xA48, 13, 0xA49, 0xA4A, 9, 0xA4B, 0xA4D, 13, 0xA4E, 0xA50, 9, + 0xA51, 0xA51, 13, 0xA52, 0xA6F, 9, 0xA70, 0xA71, 13, 0xA72, 0xA74, 9, + 0xA75, 0xA75, 13, 0xA76, 0xA80, 9, 0xA81, 0xA82, 13, 0xA83, 0xABB, 9, + 0xABC, 0xABC, 13, 0xABD, 0xAC0, 9, 0xAC1, 0xAC5, 13, 0xAC6, 0xAC6, 9, + 0xAC7, 0xAC8, 13, 0xAC9, 0xACC, 9, 0xACD, 0xACD, 13, 0xACE, 0xAE1, 9, + 0xAE2, 0xAE3, 13, 0xAE4, 0xAF0, 9, 0xAF1, 0xAF1, 7, 0xAF2, 0xAF9, 9, + 0xAFA, 0xAFF, 13, 0xB00, 0xB00, 9, 0xB01, 0xB01, 13, 0xB02, 0xB3B, 9, + 0xB3C, 0xB3C, 13, 0xB3D, 0xB3E, 9, 0xB3F, 0xB3F, 13, 0xB40, 0xB40, 9, + 0xB41, 0xB44, 13, 0xB45, 0xB4C, 9, 0xB4D, 0xB4D, 13, 0xB4E, 0xB54, 9, + 0xB55, 0xB56, 13, 0xB57, 0xB61, 9, 0xB62, 0xB63, 13, 0xB64, 0xB81, 9, + 0xB82, 0xB82, 13, 0xB83, 0xBBF, 9, 0xBC0, 0xBC0, 13, 0xBC1, 0xBCC, 9, + 0xBCD, 0xBCD, 13, 0xBCE, 0xBF2, 9, 0xBF3, 0xBF8, 14, 0xBF9, 0xBF9, 7, + 0xBFA, 0xBFA, 14, 0xBFB, 0xBFF, 9, 0xC00, 0xC00, 13, 0xC01, 0xC03, 9, + 0xC04, 0xC04, 13, 0xC05, 0xC3B, 9, 0xC3C, 0xC3C, 13, 0xC3D, 0xC3D, 9, + 0xC3E, 0xC40, 13, 0xC41, 0xC45, 9, 0xC46, 0xC48, 13, 0xC49, 0xC49, 9, + 0xC4A, 0xC4D, 13, 0xC4E, 0xC54, 9, 0xC55, 0xC56, 13, 0xC57, 0xC61, 9, + 0xC62, 0xC63, 13, 0xC64, 0xC77, 9, 0xC78, 0xC7E, 14, 0xC7F, 0xC80, 9, + 0xC81, 0xC81, 13, 0xC82, 0xCBB, 9, 0xCBC, 0xCBC, 13, 0xCBD, 0xCCB, 9, + 0xCCC, 0xCCD, 13, 0xCCE, 0xCE1, 9, 0xCE2, 0xCE3, 13, 0xCE4, 0xCFF, 9, + 0xD00, 0xD01, 13, 0xD02, 0xD3A, 9, 0xD3B, 0xD3C, 13, 0xD3D, 0xD40, 9, + 0xD41, 0xD44, 13, 0xD45, 0xD4C, 9, 0xD4D, 0xD4D, 13, 0xD4E, 0xD61, 9, + 0xD62, 0xD63, 13, 0xD64, 0xD80, 9, 0xD81, 0xD81, 13, 0xD82, 0xDC9, 9, + 0xDCA, 0xDCA, 13, 0xDCB, 0xDD1, 9, 0xDD2, 0xDD4, 13, 0xDD5, 0xDD5, 9, + 0xDD6, 0xDD6, 13, 0xDD7, 0xE30, 9, 0xE31, 0xE31, 13, 0xE32, 0xE33, 9, + 0xE34, 0xE3A, 13, 0xE3B, 0xE3E, 9, 0xE3F, 0xE3F, 7, 0xE40, 0xE46, 9, + 0xE47, 0xE4E, 13, 0xE4F, 0xEB0, 9, 0xEB1, 0xEB1, 13, 0xEB2, 0xEB3, 9, + 0xEB4, 0xEBC, 13, 0xEBD, 0xEC7, 9, 0xEC8, 0xECE, 13, 0xECF, 0xF17, 9, + 0xF18, 0xF19, 13, 0xF1A, 0xF34, 9, 0xF35, 0xF35, 13, 0xF36, 0xF36, 9, + 0xF37, 0xF37, 13, 0xF38, 0xF38, 9, 0xF39, 0xF39, 13, 0xF3A, 0xF3D, 14, + 0xF3E, 0xF70, 9, 0xF71, 0xF7E, 13, 0xF7F, 0xF7F, 9, 0xF80, 0xF84, 13, + 0xF85, 0xF85, 9, 0xF86, 0xF87, 13, 0xF88, 0xF8C, 9, 0xF8D, 0xF97, 13, + 0xF98, 0xF98, 9, 0xF99, 0xFBC, 13, 0xFBD, 0xFC5, 9, 0xFC6, 0xFC6, 13, + 0xFC7, 0x102C, 9, 0x102D, 0x1030, 13, 0x1031, 0x1031, 9, 0x1032, 0x1037, 13, + 0x1038, 0x1038, 9, 0x1039, 0x103A, 13, 0x103B, 0x103C, 9, 0x103D, 0x103E, 13, + 0x103F, 0x1057, 9, 0x1058, 0x1059, 13, 0x105A, 0x105D, 9, 0x105E, 0x1060, 13, + 0x1061, 0x1070, 9, 0x1071, 0x1074, 13, 0x1075, 0x1081, 9, 0x1082, 0x1082, 13, + 0x1083, 0x1084, 9, 0x1085, 0x1086, 13, 0x1087, 0x108C, 9, 0x108D, 0x108D, 13, + 0x108E, 0x109C, 9, 0x109D, 0x109D, 13, 0x109E, 0x135C, 9, 0x135D, 0x135F, 13, + 0x1360, 0x138F, 9, 0x1390, 0x1399, 14, 0x139A, 0x13FF, 9, 0x1400, 0x1400, 14, + 0x1401, 0x167F, 9, 0x1680, 0x1680, 22, 0x1681, 0x169A, 9, 0x169B, 0x169C, 14, + 0x169D, 0x1711, 9, 0x1712, 0x1714, 13, 0x1715, 0x1731, 9, 0x1732, 0x1733, 13, + 0x1734, 0x1751, 9, 0x1752, 0x1753, 13, 0x1754, 0x1771, 9, 0x1772, 0x1773, 13, + 0x1774, 0x17B3, 9, 0x17B4, 0x17B5, 13, 0x17B6, 0x17B6, 9, 0x17B7, 0x17BD, 13, + 0x17BE, 0x17C5, 9, 0x17C6, 0x17C6, 13, 0x17C7, 0x17C8, 9, 0x17C9, 0x17D3, 13, + 0x17D4, 0x17DA, 9, 0x17DB, 0x17DB, 7, 0x17DC, 0x17DC, 9, 0x17DD, 0x17DD, 13, + 0x17DE, 0x17EF, 9, 0x17F0, 0x17F9, 14, 0x17FA, 0x17FF, 9, 0x1800, 0x180A, 14, + 0x180B, 0x180D, 13, 0x180E, 0x180E, 3, 0x180F, 0x180F, 13, 0x1810, 0x1884, 9, + 0x1885, 0x1886, 13, 0x1887, 0x18A8, 9, 0x18A9, 0x18A9, 13, 0x18AA, 0x191F, 9, + 0x1920, 0x1922, 13, 0x1923, 0x1926, 9, 0x1927, 0x1928, 13, 0x1929, 0x1931, 9, + 0x1932, 0x1932, 13, 0x1933, 0x1938, 9, 0x1939, 0x193B, 13, 0x193C, 0x193F, 9, + 0x1940, 0x1940, 14, 0x1941, 0x1943, 9, 0x1944, 0x1945, 14, 0x1946, 0x19DD, 9, + 0x19DE, 0x19FF, 14, 0x1A00, 0x1A16, 9, 0x1A17, 0x1A18, 13, 0x1A19, 0x1A1A, 9, + 0x1A1B, 0x1A1B, 13, 0x1A1C, 0x1A55, 9, 0x1A56, 0x1A56, 13, 0x1A57, 0x1A57, 9, + 0x1A58, 0x1A5E, 13, 0x1A5F, 0x1A5F, 9, 0x1A60, 0x1A60, 13, 0x1A61, 0x1A61, 9, + 0x1A62, 0x1A62, 13, 0x1A63, 0x1A64, 9, 0x1A65, 0x1A6C, 13, 0x1A6D, 0x1A72, 9, + 0x1A73, 0x1A7C, 13, 0x1A7D, 0x1A7E, 9, 0x1A7F, 0x1A7F, 13, 0x1A80, 0x1AAF, 9, + 0x1AB0, 0x1ADD, 13, 0x1ADE, 0x1ADF, 9, 0x1AE0, 0x1AEB, 13, 0x1AEC, 0x1AFF, 9, + 0x1B00, 0x1B03, 13, 0x1B04, 0x1B33, 9, 0x1B34, 0x1B34, 13, 0x1B35, 0x1B35, 9, + 0x1B36, 0x1B3A, 13, 0x1B3B, 0x1B3B, 9, 0x1B3C, 0x1B3C, 13, 0x1B3D, 0x1B41, 9, + 0x1B42, 0x1B42, 13, 0x1B43, 0x1B6A, 9, 0x1B6B, 0x1B73, 13, 0x1B74, 0x1B7F, 9, + 0x1B80, 0x1B81, 13, 0x1B82, 0x1BA1, 9, 0x1BA2, 0x1BA5, 13, 0x1BA6, 0x1BA7, 9, + 0x1BA8, 0x1BA9, 13, 0x1BAA, 0x1BAA, 9, 0x1BAB, 0x1BAD, 13, 0x1BAE, 0x1BE5, 9, + 0x1BE6, 0x1BE6, 13, 0x1BE7, 0x1BE7, 9, 0x1BE8, 0x1BE9, 13, 0x1BEA, 0x1BEC, 9, + 0x1BED, 0x1BED, 13, 0x1BEE, 0x1BEE, 9, 0x1BEF, 0x1BF1, 13, 0x1BF2, 0x1C2B, 9, + 0x1C2C, 0x1C33, 13, 0x1C34, 0x1C35, 9, 0x1C36, 0x1C37, 13, 0x1C38, 0x1CCF, 9, + 0x1CD0, 0x1CD2, 13, 0x1CD3, 0x1CD3, 9, 0x1CD4, 0x1CE0, 13, 0x1CE1, 0x1CE1, 9, + 0x1CE2, 0x1CE8, 13, 0x1CE9, 0x1CEC, 9, 0x1CED, 0x1CED, 13, 0x1CEE, 0x1CF3, 9, + 0x1CF4, 0x1CF4, 13, 0x1CF5, 0x1CF7, 9, 0x1CF8, 0x1CF9, 13, 0x1CFA, 0x1DBF, 9, + 0x1DC0, 0x1DFF, 13, 0x1E00, 0x1FBC, 9, 0x1FBD, 0x1FBD, 14, 0x1FBE, 0x1FBE, 9, + 0x1FBF, 0x1FC1, 14, 0x1FC2, 0x1FCC, 9, 0x1FCD, 0x1FCF, 14, 0x1FD0, 0x1FDC, 9, + 0x1FDD, 0x1FDF, 14, 0x1FE0, 0x1FEC, 9, 0x1FED, 0x1FEF, 14, 0x1FF0, 0x1FFC, 9, + 0x1FFD, 0x1FFE, 14, 0x1FFF, 0x1FFF, 9, 0x2000, 0x200A, 22, 0x200B, 0x200D, 3, + 0x200E, 0x200E, 9, 0x200F, 0x200F, 17, 0x2010, 0x2027, 14, 0x2028, 0x2028, 22, + 0x2029, 0x2029, 2, 0x202A, 0x202A, 10, 0x202B, 0x202B, 18, 0x202C, 0x202C, 15, + 0x202D, 0x202D, 12, 0x202E, 0x202E, 20, 0x202F, 0x202F, 4, 0x2030, 0x2034, 7, + 0x2035, 0x2043, 14, 0x2044, 0x2044, 4, 0x2045, 0x205E, 14, 0x205F, 0x205F, 22, + 0x2060, 0x2065, 3, 0x2066, 0x2066, 11, 0x2067, 0x2067, 19, 0x2068, 0x2068, 8, + 0x2069, 0x2069, 16, 0x206A, 0x206F, 3, 0x2070, 0x2070, 5, 0x2071, 0x2073, 9, + 0x2074, 0x2079, 5, 0x207A, 0x207B, 6, 0x207C, 0x207E, 14, 0x207F, 0x207F, 9, + 0x2080, 0x2089, 5, 0x208A, 0x208B, 6, 0x208C, 0x208E, 14, 0x208F, 0x209F, 9, + 0x20A0, 0x20CF, 7, 0x20D0, 0x20F0, 13, 0x20F1, 0x20FF, 9, 0x2100, 0x2101, 14, + 0x2102, 0x2102, 9, 0x2103, 0x2106, 14, 0x2107, 0x2107, 9, 0x2108, 0x2109, 14, + 0x210A, 0x2113, 9, 0x2114, 0x2114, 14, 0x2115, 0x2115, 9, 0x2116, 0x2118, 14, + 0x2119, 0x211D, 9, 0x211E, 0x2123, 14, 0x2124, 0x2124, 9, 0x2125, 0x2125, 14, + 0x2126, 0x2126, 9, 0x2127, 0x2127, 14, 0x2128, 0x2128, 9, 0x2129, 0x2129, 14, + 0x212A, 0x212D, 9, 0x212E, 0x212E, 7, 0x212F, 0x2139, 9, 0x213A, 0x213B, 14, + 0x213C, 0x213F, 9, 0x2140, 0x2144, 14, 0x2145, 0x2149, 9, 0x214A, 0x214D, 14, + 0x214E, 0x214F, 9, 0x2150, 0x215F, 14, 0x2160, 0x2188, 9, 0x2189, 0x218B, 14, + 0x218C, 0x218F, 9, 0x2190, 0x2211, 14, 0x2212, 0x2212, 6, 0x2213, 0x2213, 7, + 0x2214, 0x2335, 14, 0x2336, 0x237A, 9, 0x237B, 0x2394, 14, 0x2395, 0x2395, 9, + 0x2396, 0x2429, 14, 0x242A, 0x243F, 9, 0x2440, 0x244A, 14, 0x244B, 0x245F, 9, + 0x2460, 0x2487, 14, 0x2488, 0x249B, 5, 0x249C, 0x24E9, 9, 0x24EA, 0x26AB, 14, + 0x26AC, 0x26AC, 9, 0x26AD, 0x27FF, 14, 0x2800, 0x28FF, 9, 0x2900, 0x2B73, 14, + 0x2B74, 0x2B75, 9, 0x2B76, 0x2BFF, 14, 0x2C00, 0x2CE4, 9, 0x2CE5, 0x2CEA, 14, + 0x2CEB, 0x2CEE, 9, 0x2CEF, 0x2CF1, 13, 0x2CF2, 0x2CF8, 9, 0x2CF9, 0x2CFF, 14, + 0x2D00, 0x2D7E, 9, 0x2D7F, 0x2D7F, 13, 0x2D80, 0x2DDF, 9, 0x2DE0, 0x2DFF, 13, + 0x2E00, 0x2E5D, 14, 0x2E5E, 0x2E7F, 9, 0x2E80, 0x2E99, 14, 0x2E9A, 0x2E9A, 9, + 0x2E9B, 0x2EF3, 14, 0x2EF4, 0x2EFF, 9, 0x2F00, 0x2FD5, 14, 0x2FD6, 0x2FEF, 9, + 0x2FF0, 0x2FFF, 14, 0x3000, 0x3000, 22, 0x3001, 0x3004, 14, 0x3005, 0x3007, 9, + 0x3008, 0x3020, 14, 0x3021, 0x3029, 9, 0x302A, 0x302D, 13, 0x302E, 0x302F, 9, + 0x3030, 0x3030, 14, 0x3031, 0x3035, 9, 0x3036, 0x3037, 14, 0x3038, 0x303C, 9, + 0x303D, 0x303F, 14, 0x3040, 0x3098, 9, 0x3099, 0x309A, 13, 0x309B, 0x309C, 14, + 0x309D, 0x309F, 9, 0x30A0, 0x30A0, 14, 0x30A1, 0x30FA, 9, 0x30FB, 0x30FB, 14, + 0x30FC, 0x31BF, 9, 0x31C0, 0x31E5, 14, 0x31E6, 0x31EE, 9, 0x31EF, 0x31EF, 14, + 0x31F0, 0x321C, 9, 0x321D, 0x321E, 14, 0x321F, 0x324F, 9, 0x3250, 0x325F, 14, + 0x3260, 0x327B, 9, 0x327C, 0x327E, 14, 0x327F, 0x32B0, 9, 0x32B1, 0x32BF, 14, + 0x32C0, 0x32CB, 9, 0x32CC, 0x32CF, 14, 0x32D0, 0x3376, 9, 0x3377, 0x337A, 14, + 0x337B, 0x33DD, 9, 0x33DE, 0x33DF, 14, 0x33E0, 0x33FE, 9, 0x33FF, 0x33FF, 14, + 0x3400, 0x4DBF, 9, 0x4DC0, 0x4DFF, 14, 0x4E00, 0xA48F, 9, 0xA490, 0xA4C6, 14, + 0xA4C7, 0xA60C, 9, 0xA60D, 0xA60F, 14, 0xA610, 0xA66E, 9, 0xA66F, 0xA672, 13, + 0xA673, 0xA673, 14, 0xA674, 0xA67D, 13, 0xA67E, 0xA67F, 14, 0xA680, 0xA69D, 9, + 0xA69E, 0xA69F, 13, 0xA6A0, 0xA6EF, 9, 0xA6F0, 0xA6F1, 13, 0xA6F2, 0xA6FF, 9, + 0xA700, 0xA721, 14, 0xA722, 0xA787, 9, 0xA788, 0xA788, 14, 0xA789, 0xA801, 9, + 0xA802, 0xA802, 13, 0xA803, 0xA805, 9, 0xA806, 0xA806, 13, 0xA807, 0xA80A, 9, + 0xA80B, 0xA80B, 13, 0xA80C, 0xA824, 9, 0xA825, 0xA826, 13, 0xA827, 0xA827, 9, + 0xA828, 0xA82B, 14, 0xA82C, 0xA82C, 13, 0xA82D, 0xA837, 9, 0xA838, 0xA839, 7, + 0xA83A, 0xA873, 9, 0xA874, 0xA877, 14, 0xA878, 0xA8C3, 9, 0xA8C4, 0xA8C5, 13, + 0xA8C6, 0xA8DF, 9, 0xA8E0, 0xA8F1, 13, 0xA8F2, 0xA8FE, 9, 0xA8FF, 0xA8FF, 13, + 0xA900, 0xA925, 9, 0xA926, 0xA92D, 13, 0xA92E, 0xA946, 9, 0xA947, 0xA951, 13, + 0xA952, 0xA97F, 9, 0xA980, 0xA982, 13, 0xA983, 0xA9B2, 9, 0xA9B3, 0xA9B3, 13, + 0xA9B4, 0xA9B5, 9, 0xA9B6, 0xA9B9, 13, 0xA9BA, 0xA9BB, 9, 0xA9BC, 0xA9BD, 13, + 0xA9BE, 0xA9E4, 9, 0xA9E5, 0xA9E5, 13, 0xA9E6, 0xAA28, 9, 0xAA29, 0xAA2E, 13, + 0xAA2F, 0xAA30, 9, 0xAA31, 0xAA32, 13, 0xAA33, 0xAA34, 9, 0xAA35, 0xAA36, 13, + 0xAA37, 0xAA42, 9, 0xAA43, 0xAA43, 13, 0xAA44, 0xAA4B, 9, 0xAA4C, 0xAA4C, 13, + 0xAA4D, 0xAA7B, 9, 0xAA7C, 0xAA7C, 13, 0xAA7D, 0xAAAF, 9, 0xAAB0, 0xAAB0, 13, + 0xAAB1, 0xAAB1, 9, 0xAAB2, 0xAAB4, 13, 0xAAB5, 0xAAB6, 9, 0xAAB7, 0xAAB8, 13, + 0xAAB9, 0xAABD, 9, 0xAABE, 0xAABF, 13, 0xAAC0, 0xAAC0, 9, 0xAAC1, 0xAAC1, 13, + 0xAAC2, 0xAAEB, 9, 0xAAEC, 0xAAED, 13, 0xAAEE, 0xAAF5, 9, 0xAAF6, 0xAAF6, 13, + 0xAAF7, 0xAB69, 9, 0xAB6A, 0xAB6B, 14, 0xAB6C, 0xABE4, 9, 0xABE5, 0xABE5, 13, + 0xABE6, 0xABE7, 9, 0xABE8, 0xABE8, 13, 0xABE9, 0xABEC, 9, 0xABED, 0xABED, 13, + 0xABEE, 0xFB1C, 9, 0xFB1D, 0xFB1D, 17, 0xFB1E, 0xFB1E, 13, 0xFB1F, 0xFB28, 17, + 0xFB29, 0xFB29, 6, 0xFB2A, 0xFB4F, 17, 0xFB50, 0xFBC2, 0, 0xFBC3, 0xFBD2, 14, + 0xFBD3, 0xFD3D, 0, 0xFD3E, 0xFD4F, 14, 0xFD50, 0xFD8F, 0, 0xFD90, 0xFD91, 14, + 0xFD92, 0xFDC7, 0, 0xFDC8, 0xFDCF, 14, 0xFDD0, 0xFDEF, 3, 0xFDF0, 0xFDFC, 0, + 0xFDFD, 0xFDFF, 14, 0xFE00, 0xFE0F, 13, 0xFE10, 0xFE19, 14, 0xFE1A, 0xFE1F, 9, + 0xFE20, 0xFE2F, 13, 0xFE30, 0xFE4F, 14, 0xFE50, 0xFE50, 4, 0xFE51, 0xFE51, 14, + 0xFE52, 0xFE52, 4, 0xFE53, 0xFE53, 9, 0xFE54, 0xFE54, 14, 0xFE55, 0xFE55, 4, + 0xFE56, 0xFE5E, 14, 0xFE5F, 0xFE5F, 7, 0xFE60, 0xFE61, 14, 0xFE62, 0xFE63, 6, + 0xFE64, 0xFE66, 14, 0xFE67, 0xFE67, 9, 0xFE68, 0xFE68, 14, 0xFE69, 0xFE6A, 7, + 0xFE6B, 0xFE6B, 14, 0xFE6C, 0xFE6F, 9, 0xFE70, 0xFEFE, 0, 0xFEFF, 0xFEFF, 3, + 0xFF00, 0xFF00, 9, 0xFF01, 0xFF02, 14, 0xFF03, 0xFF05, 7, 0xFF06, 0xFF0A, 14, + 0xFF0B, 0xFF0B, 6, 0xFF0C, 0xFF0C, 4, 0xFF0D, 0xFF0D, 6, 0xFF0E, 0xFF0F, 4, + 0xFF10, 0xFF19, 5, 0xFF1A, 0xFF1A, 4, 0xFF1B, 0xFF20, 14, 0xFF21, 0xFF3A, 9, + 0xFF3B, 0xFF40, 14, 0xFF41, 0xFF5A, 9, 0xFF5B, 0xFF65, 14, 0xFF66, 0xFFDF, 9, + 0xFFE0, 0xFFE1, 7, 0xFFE2, 0xFFE4, 14, 0xFFE5, 0xFFE6, 7, 0xFFE7, 0xFFE7, 9, + 0xFFE8, 0xFFEE, 14, 0xFFEF, 0xFFEF, 9, 0xFFF0, 0xFFF8, 3, 0xFFF9, 0xFFFD, 14, + 0xFFFE, 0xFFFF, 3, 0x10000, 0x10100, 9, 0x10101, 0x10101, 14, 0x10102, 0x1013F, 9, + 0x10140, 0x1018C, 14, 0x1018D, 0x1018F, 9, 0x10190, 0x1019C, 14, 0x1019D, 0x1019F, 9, + 0x101A0, 0x101A0, 14, 0x101A1, 0x101FC, 9, 0x101FD, 0x101FD, 13, 0x101FE, 0x102DF, 9, + 0x102E0, 0x102E0, 13, 0x102E1, 0x102FB, 5, 0x102FC, 0x10375, 9, 0x10376, 0x1037A, 13, + 0x1037B, 0x107FF, 9, 0x10800, 0x1091E, 17, 0x1091F, 0x1091F, 14, 0x10920, 0x10A00, 17, + 0x10A01, 0x10A03, 13, 0x10A04, 0x10A04, 17, 0x10A05, 0x10A06, 13, 0x10A07, 0x10A0B, 17, + 0x10A0C, 0x10A0F, 13, 0x10A10, 0x10A37, 17, 0x10A38, 0x10A3A, 13, 0x10A3B, 0x10A3E, 17, + 0x10A3F, 0x10A3F, 13, 0x10A40, 0x10AE4, 17, 0x10AE5, 0x10AE6, 13, 0x10AE7, 0x10B38, 17, + 0x10B39, 0x10B3F, 14, 0x10B40, 0x10CFF, 17, 0x10D00, 0x10D23, 0, 0x10D24, 0x10D27, 13, + 0x10D28, 0x10D2F, 0, 0x10D30, 0x10D39, 1, 0x10D3A, 0x10D3F, 0, 0x10D40, 0x10D49, 1, + 0x10D4A, 0x10D68, 17, 0x10D69, 0x10D6D, 13, 0x10D6E, 0x10D6E, 14, 0x10D6F, 0x10E5F, 17, + 0x10E60, 0x10E7E, 1, 0x10E7F, 0x10EAA, 17, 0x10EAB, 0x10EAC, 13, 0x10EAD, 0x10EBF, 17, + 0x10EC0, 0x10ECF, 0, 0x10ED0, 0x10ED8, 14, 0x10ED9, 0x10EF9, 0, 0x10EFA, 0x10EFF, 13, + 0x10F00, 0x10F2F, 17, 0x10F30, 0x10F45, 0, 0x10F46, 0x10F50, 13, 0x10F51, 0x10F6F, 0, + 0x10F70, 0x10F81, 17, 0x10F82, 0x10F85, 13, 0x10F86, 0x10FFF, 17, 0x11000, 0x11000, 9, + 0x11001, 0x11001, 13, 0x11002, 0x11037, 9, 0x11038, 0x11046, 13, 0x11047, 0x11051, 9, + 0x11052, 0x11065, 14, 0x11066, 0x1106F, 9, 0x11070, 0x11070, 13, 0x11071, 0x11072, 9, + 0x11073, 0x11074, 13, 0x11075, 0x1107E, 9, 0x1107F, 0x11081, 13, 0x11082, 0x110B2, 9, + 0x110B3, 0x110B6, 13, 0x110B7, 0x110B8, 9, 0x110B9, 0x110BA, 13, 0x110BB, 0x110C1, 9, + 0x110C2, 0x110C2, 13, 0x110C3, 0x110FF, 9, 0x11100, 0x11102, 13, 0x11103, 0x11126, 9, + 0x11127, 0x1112B, 13, 0x1112C, 0x1112C, 9, 0x1112D, 0x11134, 13, 0x11135, 0x11172, 9, + 0x11173, 0x11173, 13, 0x11174, 0x1117F, 9, 0x11180, 0x11181, 13, 0x11182, 0x111B5, 9, + 0x111B6, 0x111BE, 13, 0x111BF, 0x111C8, 9, 0x111C9, 0x111CC, 13, 0x111CD, 0x111CE, 9, + 0x111CF, 0x111CF, 13, 0x111D0, 0x1122E, 9, 0x1122F, 0x11231, 13, 0x11232, 0x11233, 9, + 0x11234, 0x11234, 13, 0x11235, 0x11235, 9, 0x11236, 0x11237, 13, 0x11238, 0x1123D, 9, + 0x1123E, 0x1123E, 13, 0x1123F, 0x11240, 9, 0x11241, 0x11241, 13, 0x11242, 0x112DE, 9, + 0x112DF, 0x112DF, 13, 0x112E0, 0x112E2, 9, 0x112E3, 0x112EA, 13, 0x112EB, 0x112FF, 9, + 0x11300, 0x11301, 13, 0x11302, 0x1133A, 9, 0x1133B, 0x1133C, 13, 0x1133D, 0x1133F, 9, + 0x11340, 0x11340, 13, 0x11341, 0x11365, 9, 0x11366, 0x1136C, 13, 0x1136D, 0x1136F, 9, + 0x11370, 0x11374, 13, 0x11375, 0x113BA, 9, 0x113BB, 0x113C0, 13, 0x113C1, 0x113CD, 9, + 0x113CE, 0x113CE, 13, 0x113CF, 0x113CF, 9, 0x113D0, 0x113D0, 13, 0x113D1, 0x113D1, 9, + 0x113D2, 0x113D2, 13, 0x113D3, 0x113E0, 9, 0x113E1, 0x113E2, 13, 0x113E3, 0x11437, 9, + 0x11438, 0x1143F, 13, 0x11440, 0x11441, 9, 0x11442, 0x11444, 13, 0x11445, 0x11445, 9, + 0x11446, 0x11446, 13, 0x11447, 0x1145D, 9, 0x1145E, 0x1145E, 13, 0x1145F, 0x114B2, 9, + 0x114B3, 0x114B8, 13, 0x114B9, 0x114B9, 9, 0x114BA, 0x114BA, 13, 0x114BB, 0x114BE, 9, + 0x114BF, 0x114C0, 13, 0x114C1, 0x114C1, 9, 0x114C2, 0x114C3, 13, 0x114C4, 0x115B1, 9, + 0x115B2, 0x115B5, 13, 0x115B6, 0x115BB, 9, 0x115BC, 0x115BD, 13, 0x115BE, 0x115BE, 9, + 0x115BF, 0x115C0, 13, 0x115C1, 0x115DB, 9, 0x115DC, 0x115DD, 13, 0x115DE, 0x11632, 9, + 0x11633, 0x1163A, 13, 0x1163B, 0x1163C, 9, 0x1163D, 0x1163D, 13, 0x1163E, 0x1163E, 9, + 0x1163F, 0x11640, 13, 0x11641, 0x1165F, 9, 0x11660, 0x1166C, 14, 0x1166D, 0x116AA, 9, + 0x116AB, 0x116AB, 13, 0x116AC, 0x116AC, 9, 0x116AD, 0x116AD, 13, 0x116AE, 0x116AF, 9, + 0x116B0, 0x116B5, 13, 0x116B6, 0x116B6, 9, 0x116B7, 0x116B7, 13, 0x116B8, 0x1171C, 9, + 0x1171D, 0x1171D, 13, 0x1171E, 0x1171E, 9, 0x1171F, 0x1171F, 13, 0x11720, 0x11721, 9, + 0x11722, 0x11725, 13, 0x11726, 0x11726, 9, 0x11727, 0x1172B, 13, 0x1172C, 0x1182E, 9, + 0x1182F, 0x11837, 13, 0x11838, 0x11838, 9, 0x11839, 0x1183A, 13, 0x1183B, 0x1193A, 9, + 0x1193B, 0x1193C, 13, 0x1193D, 0x1193D, 9, 0x1193E, 0x1193E, 13, 0x1193F, 0x11942, 9, + 0x11943, 0x11943, 13, 0x11944, 0x119D3, 9, 0x119D4, 0x119D7, 13, 0x119D8, 0x119D9, 9, + 0x119DA, 0x119DB, 13, 0x119DC, 0x119DF, 9, 0x119E0, 0x119E0, 13, 0x119E1, 0x11A00, 9, + 0x11A01, 0x11A06, 13, 0x11A07, 0x11A08, 9, 0x11A09, 0x11A0A, 13, 0x11A0B, 0x11A32, 9, + 0x11A33, 0x11A38, 13, 0x11A39, 0x11A3A, 9, 0x11A3B, 0x11A3E, 13, 0x11A3F, 0x11A46, 9, + 0x11A47, 0x11A47, 13, 0x11A48, 0x11A50, 9, 0x11A51, 0x11A56, 13, 0x11A57, 0x11A58, 9, + 0x11A59, 0x11A5B, 13, 0x11A5C, 0x11A89, 9, 0x11A8A, 0x11A96, 13, 0x11A97, 0x11A97, 9, + 0x11A98, 0x11A99, 13, 0x11A9A, 0x11B5F, 9, 0x11B60, 0x11B60, 13, 0x11B61, 0x11B61, 9, + 0x11B62, 0x11B64, 13, 0x11B65, 0x11B65, 9, 0x11B66, 0x11B66, 13, 0x11B67, 0x11C2F, 9, + 0x11C30, 0x11C36, 13, 0x11C37, 0x11C37, 9, 0x11C38, 0x11C3D, 13, 0x11C3E, 0x11C91, 9, + 0x11C92, 0x11CA7, 13, 0x11CA8, 0x11CA9, 9, 0x11CAA, 0x11CB0, 13, 0x11CB1, 0x11CB1, 9, + 0x11CB2, 0x11CB3, 13, 0x11CB4, 0x11CB4, 9, 0x11CB5, 0x11CB6, 13, 0x11CB7, 0x11D30, 9, + 0x11D31, 0x11D36, 13, 0x11D37, 0x11D39, 9, 0x11D3A, 0x11D3A, 13, 0x11D3B, 0x11D3B, 9, + 0x11D3C, 0x11D3D, 13, 0x11D3E, 0x11D3E, 9, 0x11D3F, 0x11D45, 13, 0x11D46, 0x11D46, 9, + 0x11D47, 0x11D47, 13, 0x11D48, 0x11D8F, 9, 0x11D90, 0x11D91, 13, 0x11D92, 0x11D94, 9, + 0x11D95, 0x11D95, 13, 0x11D96, 0x11D96, 9, 0x11D97, 0x11D97, 13, 0x11D98, 0x11EF2, 9, + 0x11EF3, 0x11EF4, 13, 0x11EF5, 0x11EFF, 9, 0x11F00, 0x11F01, 13, 0x11F02, 0x11F35, 9, + 0x11F36, 0x11F3A, 13, 0x11F3B, 0x11F3F, 9, 0x11F40, 0x11F40, 13, 0x11F41, 0x11F41, 9, + 0x11F42, 0x11F42, 13, 0x11F43, 0x11F59, 9, 0x11F5A, 0x11F5A, 13, 0x11F5B, 0x11FD4, 9, + 0x11FD5, 0x11FDC, 14, 0x11FDD, 0x11FE0, 7, 0x11FE1, 0x11FF1, 14, 0x11FF2, 0x1343F, 9, + 0x13440, 0x13440, 13, 0x13441, 0x13446, 9, 0x13447, 0x13455, 13, 0x13456, 0x1611D, 9, + 0x1611E, 0x16129, 13, 0x1612A, 0x1612C, 9, 0x1612D, 0x1612F, 13, 0x16130, 0x16AEF, 9, + 0x16AF0, 0x16AF4, 13, 0x16AF5, 0x16B2F, 9, 0x16B30, 0x16B36, 13, 0x16B37, 0x16F4E, 9, + 0x16F4F, 0x16F4F, 13, 0x16F50, 0x16F8E, 9, 0x16F8F, 0x16F92, 13, 0x16F93, 0x16FE1, 9, + 0x16FE2, 0x16FE2, 14, 0x16FE3, 0x16FE3, 9, 0x16FE4, 0x16FE4, 13, 0x16FE5, 0x1BC9C, 9, + 0x1BC9D, 0x1BC9E, 13, 0x1BC9F, 0x1BC9F, 9, 0x1BCA0, 0x1BCA3, 3, 0x1BCA4, 0x1CBFF, 9, + 0x1CC00, 0x1CCD5, 14, 0x1CCD6, 0x1CCEF, 9, 0x1CCF0, 0x1CCF9, 5, 0x1CCFA, 0x1CCFC, 14, + 0x1CCFD, 0x1CCFF, 9, 0x1CD00, 0x1CEB3, 14, 0x1CEB4, 0x1CEB9, 9, 0x1CEBA, 0x1CED0, 14, + 0x1CED1, 0x1CEDF, 9, 0x1CEE0, 0x1CEF0, 14, 0x1CEF1, 0x1CEFF, 9, 0x1CF00, 0x1CF2D, 13, + 0x1CF2E, 0x1CF2F, 9, 0x1CF30, 0x1CF46, 13, 0x1CF47, 0x1D166, 9, 0x1D167, 0x1D169, 13, + 0x1D16A, 0x1D172, 9, 0x1D173, 0x1D17A, 3, 0x1D17B, 0x1D182, 13, 0x1D183, 0x1D184, 9, + 0x1D185, 0x1D18B, 13, 0x1D18C, 0x1D1A9, 9, 0x1D1AA, 0x1D1AD, 13, 0x1D1AE, 0x1D1E8, 9, + 0x1D1E9, 0x1D1EA, 14, 0x1D1EB, 0x1D1FF, 9, 0x1D200, 0x1D241, 14, 0x1D242, 0x1D244, 13, + 0x1D245, 0x1D245, 14, 0x1D246, 0x1D2FF, 9, 0x1D300, 0x1D356, 14, 0x1D357, 0x1D6C0, 9, + 0x1D6C1, 0x1D6C1, 14, 0x1D6C2, 0x1D6DA, 9, 0x1D6DB, 0x1D6DB, 14, 0x1D6DC, 0x1D6FA, 9, + 0x1D6FB, 0x1D6FB, 14, 0x1D6FC, 0x1D714, 9, 0x1D715, 0x1D715, 14, 0x1D716, 0x1D734, 9, + 0x1D735, 0x1D735, 14, 0x1D736, 0x1D74E, 9, 0x1D74F, 0x1D74F, 14, 0x1D750, 0x1D76E, 9, + 0x1D76F, 0x1D76F, 14, 0x1D770, 0x1D788, 9, 0x1D789, 0x1D789, 14, 0x1D78A, 0x1D7A8, 9, + 0x1D7A9, 0x1D7A9, 14, 0x1D7AA, 0x1D7C2, 9, 0x1D7C3, 0x1D7C3, 14, 0x1D7C4, 0x1D7CD, 9, + 0x1D7CE, 0x1D7FF, 5, 0x1D800, 0x1D9FF, 9, 0x1DA00, 0x1DA36, 13, 0x1DA37, 0x1DA3A, 9, + 0x1DA3B, 0x1DA6C, 13, 0x1DA6D, 0x1DA74, 9, 0x1DA75, 0x1DA75, 13, 0x1DA76, 0x1DA83, 9, + 0x1DA84, 0x1DA84, 13, 0x1DA85, 0x1DA9A, 9, 0x1DA9B, 0x1DA9F, 13, 0x1DAA0, 0x1DAA0, 9, + 0x1DAA1, 0x1DAAF, 13, 0x1DAB0, 0x1DFFF, 9, 0x1E000, 0x1E006, 13, 0x1E007, 0x1E007, 9, + 0x1E008, 0x1E018, 13, 0x1E019, 0x1E01A, 9, 0x1E01B, 0x1E021, 13, 0x1E022, 0x1E022, 9, + 0x1E023, 0x1E024, 13, 0x1E025, 0x1E025, 9, 0x1E026, 0x1E02A, 13, 0x1E02B, 0x1E08E, 9, + 0x1E08F, 0x1E08F, 13, 0x1E090, 0x1E12F, 9, 0x1E130, 0x1E136, 13, 0x1E137, 0x1E2AD, 9, + 0x1E2AE, 0x1E2AE, 13, 0x1E2AF, 0x1E2EB, 9, 0x1E2EC, 0x1E2EF, 13, 0x1E2F0, 0x1E2FE, 9, + 0x1E2FF, 0x1E2FF, 7, 0x1E300, 0x1E4EB, 9, 0x1E4EC, 0x1E4EF, 13, 0x1E4F0, 0x1E5ED, 9, + 0x1E5EE, 0x1E5EF, 13, 0x1E5F0, 0x1E6E2, 9, 0x1E6E3, 0x1E6E3, 13, 0x1E6E4, 0x1E6E5, 9, + 0x1E6E6, 0x1E6E6, 13, 0x1E6E7, 0x1E6ED, 9, 0x1E6EE, 0x1E6EF, 13, 0x1E6F0, 0x1E6F4, 9, + 0x1E6F5, 0x1E6F5, 13, 0x1E6F6, 0x1E7FF, 9, 0x1E800, 0x1E8CF, 17, 0x1E8D0, 0x1E8D6, 13, + 0x1E8D7, 0x1E943, 17, 0x1E944, 0x1E94A, 13, 0x1E94B, 0x1EC6F, 17, 0x1EC70, 0x1ECBF, 0, + 0x1ECC0, 0x1ECFF, 17, 0x1ED00, 0x1ED4F, 0, 0x1ED50, 0x1EDFF, 17, 0x1EE00, 0x1EEEF, 0, + 0x1EEF0, 0x1EEF1, 14, 0x1EEF2, 0x1EEFF, 0, 0x1EF00, 0x1EFFF, 17, 0x1F000, 0x1F02B, 14, + 0x1F02C, 0x1F02F, 9, 0x1F030, 0x1F093, 14, 0x1F094, 0x1F09F, 9, 0x1F0A0, 0x1F0AE, 14, + 0x1F0AF, 0x1F0B0, 9, 0x1F0B1, 0x1F0BF, 14, 0x1F0C0, 0x1F0C0, 9, 0x1F0C1, 0x1F0CF, 14, + 0x1F0D0, 0x1F0D0, 9, 0x1F0D1, 0x1F0F5, 14, 0x1F0F6, 0x1F0FF, 9, 0x1F100, 0x1F10A, 5, + 0x1F10B, 0x1F10F, 14, 0x1F110, 0x1F12E, 9, 0x1F12F, 0x1F12F, 14, 0x1F130, 0x1F169, 9, + 0x1F16A, 0x1F16F, 14, 0x1F170, 0x1F1AC, 9, 0x1F1AD, 0x1F1AD, 14, 0x1F1AE, 0x1F25F, 9, + 0x1F260, 0x1F265, 14, 0x1F266, 0x1F2FF, 9, 0x1F300, 0x1F6D8, 14, 0x1F6D9, 0x1F6DB, 9, + 0x1F6DC, 0x1F6EC, 14, 0x1F6ED, 0x1F6EF, 9, 0x1F6F0, 0x1F6FC, 14, 0x1F6FD, 0x1F6FF, 9, + 0x1F700, 0x1F7D9, 14, 0x1F7DA, 0x1F7DF, 9, 0x1F7E0, 0x1F7EB, 14, 0x1F7EC, 0x1F7EF, 9, + 0x1F7F0, 0x1F7F0, 14, 0x1F7F1, 0x1F7FF, 9, 0x1F800, 0x1F80B, 14, 0x1F80C, 0x1F80F, 9, + 0x1F810, 0x1F847, 14, 0x1F848, 0x1F84F, 9, 0x1F850, 0x1F859, 14, 0x1F85A, 0x1F85F, 9, + 0x1F860, 0x1F887, 14, 0x1F888, 0x1F88F, 9, 0x1F890, 0x1F8AD, 14, 0x1F8AE, 0x1F8AF, 9, + 0x1F8B0, 0x1F8BB, 14, 0x1F8BC, 0x1F8BF, 9, 0x1F8C0, 0x1F8C1, 14, 0x1F8C2, 0x1F8CF, 9, + 0x1F8D0, 0x1F8D8, 14, 0x1F8D9, 0x1F8FF, 9, 0x1F900, 0x1FA57, 14, 0x1FA58, 0x1FA5F, 9, + 0x1FA60, 0x1FA6D, 14, 0x1FA6E, 0x1FA6F, 9, 0x1FA70, 0x1FA7C, 14, 0x1FA7D, 0x1FA7F, 9, + 0x1FA80, 0x1FA8A, 14, 0x1FA8B, 0x1FA8D, 9, 0x1FA8E, 0x1FAC6, 14, 0x1FAC7, 0x1FAC7, 9, + 0x1FAC8, 0x1FAC8, 14, 0x1FAC9, 0x1FACC, 9, 0x1FACD, 0x1FADC, 14, 0x1FADD, 0x1FADE, 9, + 0x1FADF, 0x1FAEA, 14, 0x1FAEB, 0x1FAEE, 9, 0x1FAEF, 0x1FAF8, 14, 0x1FAF9, 0x1FAFF, 9, + 0x1FB00, 0x1FB92, 14, 0x1FB93, 0x1FB93, 9, 0x1FB94, 0x1FBEF, 14, 0x1FBF0, 0x1FBF9, 5, + 0x1FBFA, 0x1FBFA, 14, 0x1FBFB, 0x1FFFD, 9, 0x1FFFE, 0x1FFFF, 3, 0x20000, 0x2FFFD, 9, + 0x2FFFE, 0x2FFFF, 3, 0x30000, 0x3FFFD, 9, 0x3FFFE, 0x3FFFF, 3, 0x40000, 0x4FFFD, 9, + 0x4FFFE, 0x4FFFF, 3, 0x50000, 0x5FFFD, 9, 0x5FFFE, 0x5FFFF, 3, 0x60000, 0x6FFFD, 9, + 0x6FFFE, 0x6FFFF, 3, 0x70000, 0x7FFFD, 9, 0x7FFFE, 0x7FFFF, 3, 0x80000, 0x8FFFD, 9, + 0x8FFFE, 0x8FFFF, 3, 0x90000, 0x9FFFD, 9, 0x9FFFE, 0x9FFFF, 3, 0xA0000, 0xAFFFD, 9, + 0xAFFFE, 0xAFFFF, 3, 0xB0000, 0xBFFFD, 9, 0xBFFFE, 0xBFFFF, 3, 0xC0000, 0xCFFFD, 9, + 0xCFFFE, 0xCFFFF, 3, 0xD0000, 0xDFFFD, 9, 0xDFFFE, 0xE00FF, 3, 0xE0100, 0xE01EF, 13, + 0xE01F0, 0xE0FFF, 3, 0xE1000, 0xEFFFD, 9, 0xEFFFE, 0xEFFFF, 3, 0xF0000, 0xFFFFD, 9, + 0xFFFFE, 0xFFFFF, 3, 0x100000, 0x10FFFD, 9, 0x10FFFE, 0x10FFFF, 3, + }; + + private static final UnicodeSet[] VALUE_SETS = buildValueSets(); + + static boolean isPropertyAlias(String alias) { + String loose = looseName(alias); + if (loose == null) return false; + for (String candidate : PROPERTY_ALIASES) { + if (candidate.equals(loose)) return true; + } + return false; + } + + static UnicodeSet valueSet(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : VALUE_SETS[index]; + } + + static String shortValue(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : SHORT_VALUES[index]; + } + + static String canonicalValue(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : LONG_VALUES[index]; + } + + static String[] canonicalValues() { + return LONG_VALUES.clone(); + } + + private static int valueIndex(String alias) { + String loose = looseName(alias); + if (loose == null) return -1; + for (int i = 0; i < VALUE_ALIASES.length; i++) { + if (VALUE_ALIASES[i].equals(loose)) return VALUE_ALIAS_INDEX[i]; + } + return -1; + } + + private static String looseName(String name) { + if (name == null) return null; + StringBuilder loose = new StringBuilder(name.length()); + for (int i = 0; i < name.length(); i++) { + char character = name.charAt(i); + if (character == '_' || character == '-' || Character.isWhitespace(character)) continue; + loose.append(Character.toLowerCase(character)); + } + return loose.toString(); + } + + private static UnicodeSet[] buildValueSets() { + UnicodeSet[] sets = new UnicodeSet[LONG_VALUES.length]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + for (int i = 0; i < RANGES.length; i += 3) { + sets[RANGES[i + 2]].add(RANGES[i], RANGES[i + 1]); + } + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + + private PerlUnicodeBidiClassData() { + } +} diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index 1ae52ce5b0..f2ee514a9b 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -959,6 +959,18 @@ && isCanonicalCombiningClassProperty(alias.substring(0, assignment))) { } return combiningClass; } + if (assignment > 0 && assignment < alias.length() - 1 + && PerlUnicodeBidiClassData.isPropertyAlias( + alias.substring(0, assignment))) { + UnicodeSet bidiClass = PerlUnicodeBidiClassData.valueSet( + alias.substring(assignment + 1)); + if (bidiClass == null) { + throw new IllegalArgumentException( + "Unsupported Bidi_Class value: " + + alias.substring(assignment + 1).trim()); + } + return bidiClass; + } if (assignment > 0 && assignment < alias.length() - 1) { Boolean value = perlBooleanPropertyValue(alias.substring(assignment + 1)); if (value != null) { diff --git a/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeBidiClassDataTest.java b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeBidiClassDataTest.java new file mode 100644 index 0000000000..84845b5814 --- /dev/null +++ b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeBidiClassDataTest.java @@ -0,0 +1,106 @@ +package org.perlonjava.runtime.regex; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import com.ibm.icu.text.UnicodeSet; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; + +@Tag("unit") +class PerlUnicodeBidiClassDataTest { + private static final String[][] VALUES = { + {"AL", "Arabic_Letter", "0608"}, + {"AN", "Arabic_Number", "0600"}, + {"B", "Paragraph_Separator", "000A"}, + {"BN", "Boundary_Neutral", "0000"}, + {"CS", "Common_Separator", "002C"}, + {"EN", "European_Number", "0030"}, + {"ES", "European_Separator", "002B"}, + {"ET", "European_Terminator", "0023"}, + {"FSI", "First_Strong_Isolate", "2068"}, + {"L", "Left_To_Right", "0041"}, + {"LRE", "Left_To_Right_Embedding", "202A"}, + {"LRI", "Left_To_Right_Isolate", "2066"}, + {"LRO", "Left_To_Right_Override", "202D"}, + {"NSM", "Nonspacing_Mark", "0300"}, + {"ON", "Other_Neutral", "0021"}, + {"PDF", "Pop_Directional_Format", "202C"}, + {"PDI", "Pop_Directional_Isolate", "2069"}, + {"R", "Right_To_Left", "05BE"}, + {"RLE", "Right_To_Left_Embedding", "202B"}, + {"RLI", "Right_To_Left_Isolate", "2067"}, + {"RLO", "Right_To_Left_Override", "202E"}, + {"S", "Segment_Separator", "0009"}, + {"WS", "White_Space", "000C"}, + }; + + @Test + void usesChecksumPinnedPerl544Unicode17Sources() { + assertEquals("17.0.0", PerlUnicodeBidiClassData.UNICODE_VERSION); + assertEquals("4867b4b7f0731ed1bfcd34cc6251211ff1542541fce0734b6fbda139ee80b3a4", + PerlUnicodeBidiClassData.DBIDI_CLASS_SHA256); + assertEquals("670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01", + PerlUnicodeBidiClassData.PROP_VALUE_ALIASES_SHA256); + assertEquals("4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb", + PerlUnicodeBidiClassData.PROPERTY_ALIASES_SHA256); + } + + @Test + void acceptsPinnedPropertyAndLooseValueAliases() { + assertTrue(PerlUnicodeBidiClassData.isPropertyAlias("bc")); + assertTrue(PerlUnicodeBidiClassData.isPropertyAlias("Bidi Class")); + assertTrue(PerlUnicodeBidiClassData.isPropertyAlias("BIDI-CLASS")); + assertFalse(PerlUnicodeBidiClassData.isPropertyAlias("Bidi Mirrored")); + + UnicodeSet arabicLetter = PerlUnicodeBidiClassData.valueSet("AL"); + assertSame(arabicLetter, PerlUnicodeBidiClassData.valueSet("Arabic_Letter")); + assertSame(arabicLetter, PerlUnicodeBidiClassData.valueSet("a-r_a b i c letter")); + assertEquals("AL", PerlUnicodeBidiClassData.shortValue("arabic letter")); + assertEquals("Arabic_Letter", PerlUnicodeBidiClassData.canonicalValue("al")); + assertNull(PerlUnicodeBidiClassData.valueSet("not a bidi class")); + assertNull(PerlUnicodeBidiClassData.canonicalValue(null)); + } + + @Test + void retainsEveryPinnedValueIncludingDirectionalControls() { + assertEquals(23, PerlUnicodeBidiClassData.canonicalValues().length); + for (String[] value : VALUES) { + UnicodeSet shortSet = PerlUnicodeBidiClassData.valueSet(value[0]); + assertNotNull(shortSet, value[0]); + assertTrue(shortSet.isFrozen(), value[0]); + assertSame(shortSet, PerlUnicodeBidiClassData.valueSet(value[1]), value[0]); + assertTrue(shortSet.contains(Integer.parseInt(value[2], 16)), value[0]); + } + } + + @Test + void appliesOrderedMissingDefaultsBeforeExplicitRanges() { + assertTrue(PerlUnicodeBidiClassData.valueSet("R").contains(0x0590)); + assertTrue(PerlUnicodeBidiClassData.valueSet("AL").contains(0x088B)); + assertTrue(PerlUnicodeBidiClassData.valueSet("ET").contains(0x20CF)); + assertTrue(PerlUnicodeBidiClassData.valueSet("L").contains(0x0378)); + assertTrue(PerlUnicodeBidiClassData.valueSet("BN").contains(0x10FFFF)); + } + + @Test + void formsOneDisjointPartitionOfAllUnicodeCodePoints() { + String[] values = PerlUnicodeBidiClassData.canonicalValues(); + UnicodeSet union = new UnicodeSet(); + for (int i = 0; i < values.length; i++) { + UnicodeSet current = PerlUnicodeBidiClassData.valueSet(values[i]); + union.addAll(current); + for (int j = i + 1; j < values.length; j++) { + UnicodeSet overlap = new UnicodeSet(current) + .retainAll(PerlUnicodeBidiClassData.valueSet(values[j])); + assertTrue(overlap.isEmpty(), values[i] + " overlaps " + values[j]); + } + } + assertEquals(0x110000, union.size()); + assertTrue(union.contains(0, 0x10FFFF)); + } +} diff --git a/src/test/resources/unit/regex/unicode_bidi_class_pinned.t b/src/test/resources/unit/regex/unicode_bidi_class_pinned.t new file mode 100644 index 0000000000..a3f3ac3e45 --- /dev/null +++ b/src/test/resources/unit/regex/unicode_bidi_class_pinned.t @@ -0,0 +1,67 @@ +use strict; +use warnings; +use Test::More; + +my $bidi_class_probe = eval q{qr/\p{bc=L}/}; +if (!defined $bidi_class_probe) { + plan skip_all => 'pinned Bidi_Class data awaits resolver integration'; +} + +my @values = ( + ['AL', 'Arabic_Letter', 0x0608], + ['AN', 'Arabic_Number', 0x0600], + ['B', 'Paragraph_Separator', 0x000A], + ['BN', 'Boundary_Neutral', 0x0000], + ['CS', 'Common_Separator', 0x002C], + ['EN', 'European_Number', 0x0030], + ['ES', 'European_Separator', 0x002B], + ['ET', 'European_Terminator', 0x0023], + ['FSI', 'First_Strong_Isolate', 0x2068], + ['L', 'Left_To_Right', 0x0041], + ['LRE', 'Left_To_Right_Embedding', 0x202A], + ['LRI', 'Left_To_Right_Isolate', 0x2066], + ['LRO', 'Left_To_Right_Override', 0x202D], + ['NSM', 'Nonspacing_Mark', 0x0300], + ['ON', 'Other_Neutral', 0x0021], + ['PDF', 'Pop_Directional_Format', 0x202C], + ['PDI', 'Pop_Directional_Isolate', 0x2069], + ['R', 'Right_To_Left', 0x05BE], + ['RLE', 'Right_To_Left_Embedding', 0x202B], + ['RLI', 'Right_To_Left_Isolate', 0x2067], + ['RLO', 'Right_To_Left_Override', 0x202E], + ['S', 'Segment_Separator', 0x0009], + ['WS', 'White_Space', 0x000C], +); + +for my $value (@values) { + my ($short, $long, $code_point) = @$value; + my $character = chr($code_point); + my $short_pattern = eval "qr/\\p{bc=$short}/"; + ok(defined $short_pattern, "short value $short compiles") + or diag($@); + like($character, $short_pattern, "short value $short matches representative"); + + my $long_pattern = eval "qr/\\p{Bidi_Class=$long}/"; + ok(defined $long_pattern, "long value $long compiles") + or diag($@); + like($character, $long_pattern, "long value $long matches representative"); +} + +my $loose = eval q{qr/\p{bidi class = arabic letter}/}; +ok(defined $loose, 'loose property and value aliases compile') or diag($@); +like(chr(0x0608), $loose, 'loose property and value aliases match'); + +my @defaults = ( + [0x0590, 'R', 'Hebrew missing default is Right_To_Left'], + [0x088B, 'AL', 'Arabic missing default is Arabic_Letter'], + [0x20CF, 'ET', 'currency missing default is European_Terminator'], + [0x0378, 'L', 'general missing default is Left_To_Right'], + [0x10FFFF, 'BN', 'noncharacter explicit value overrides defaults'], +); +for my $default (@defaults) { + my ($code_point, $value, $description) = @$default; + my $pattern = eval "qr/\\p{bc=$value}/"; + like(chr($code_point), $pattern, $description); +} + +done_testing(); From 3239a79fd743371d84068dd6cb8685a2aa26ba19 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:33 +0200 Subject: [PATCH 29/60] fix(regex): implement Perl vertical whitespace escapes in Joni (#1034) Squashed integration of PR #1034 at 4755b5518. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 29 ++++-- docs/reference/feature-matrix.md | 1 + .../unit/regex/vertical_whitespace_escape.t | 31 +++++++ third_party/joni/src/org/joni/Lexer.java | 88 ++++++++++++++++++ .../joni/test/TestPerlVerticalWhitespace.java | 90 +++++++++++++++++++ 5 files changed, 230 insertions(+), 9 deletions(-) create mode 100644 src/test/resources/unit/regex/vertical_whitespace_escape.t create mode 100644 third_party/joni/test/org/joni/test/TestPerlVerticalWhitespace.java diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 95ae8c158c..5440afd580 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,7 +224,7 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1032, stacked +The published integration stack is preserved through draft PR #1033, stacked on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the @@ -236,8 +236,9 @@ progression and pinned Perl 5.44 Unicode 17.0 Age properties. PR #1030 adds binary `ASCII_Hex_Digit` values, pinned General_Category sets, and exact native line boundaries. PR #1031 adds pinned Canonical_Combining_Class sets and valid empty-property rendering. PR #1032 integrates native numeric escapes through -U+10FFFF. The new WIP adds pinned Bidi_Class sets while Decomposition_Type, -East_Asian_Width, and vertical-whitespace slices advance independently. +U+10FFFF; PR #1033 adds pinned Bidi_Class sets. The new WIP integrates native +vertical-whitespace escapes while Decomposition_Type, East_Asian_Width, and +Numeric_Value slices advance independently. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -320,6 +321,13 @@ combined Unicode/property/boundary smoke is 345/345 per backend. Chunk 01 gains 736 assertions to 34,252/41,843 identically on both execution backends with no numbered regression, raising current generated evidence to 342,016/407,367. +Native Joni `\v` now matches Perl's seven vertical-whitespace code points and +`\V` matches their complement, both directly and inside character classes, +without changing non-Perl Joni syntax behavior. The focused oracle passes +92/92 on system Perl, JVM, and interpreter. Unchanged `reg_posixcc.t` improves +from 2,052/2,560 to 2,560/2,560 on both execution backends with zero numbered +regressions, closing its entire 508-assertion Joni gap. + Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while ordinary and substitution callbacks execute, without misclassifying escaped or @@ -570,6 +578,9 @@ is retained for now. partition with loose aliases and ordered missing defaults. The focused oracle passes 99/99 and chunk 01 gains 736 assertions to 34,252/41,843 on both backends with zero numbered regressions. + - [x] Integrated native Perl `\v`/`\V` dispatch inside and outside character + classes (`1eff1db97`, integrated as `6328935cd`). The focused oracle passes + 92/92 and unchanged `reg_posixcc.t` passes 2,560/2,560 on both backends. - [ ] Close the remaining property failures before marking Phase 3 complete. - [x] Phase 4: Runtime source and diagnostics (2026-08-17; semantic gate complete at 550/555) @@ -612,15 +623,15 @@ is retained for now. ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1032. Publish the validated - Bidi_Class slice, then integrate native vertical-whitespace commit - `1eff1db97` and the completed Decomposition_Type data handoff in separate - focused WIP PRs. +1. Land review-ready PR #1024 and draft PRs #1025–#1033. Publish the validated + native vertical-whitespace slice, then integrate the completed + Decomposition_Type data handoff in a separate focused WIP PR. 2. Preserve the now-complete native Joni boundary corpus at 239,866/239,866: sentence chunk 05, line chunks 06–09, and word chunk 10 must remain exact on JVM and interpreter while property and parser work continues. -3. Integrate the independently generated Decomposition_Type and - East_Asian_Width slices, then continue the measured residual order: binary +3. Integrate the independently generated Decomposition_Type, + East_Asian_Width, and Numeric_Value slices, then continue the measured + residual order: binary property values, Block, Script/Script_Extensions, Numeric_Value, Joining_Group, and break-property values. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index f900c6d84b..8ca13fc07e 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -371,6 +371,7 @@ my @copy = @{$z}; # ERROR - ✅ **Inline comments**: `(?#comment)` in regex is implemented. - ✅ **caret modifier**: `(?^` embedded pattern-match modifier, shorthand equivalent to "d-imnsx". - ✅ **\b inside character class**: `[\b]` is supported in regex. +- ✅ **Vertical whitespace escapes**: Native Joni `\v` matches U+000A..U+000D, U+0085, U+2028, and U+2029; `\V` matches the complement. Direct and character-class forms match Perl, and `reg_posixcc.t` passes 2,560/2,560 on both execution backends. - ✅ **Unicode boundary assertions**: `\b{gcb}`, `\b{sb}`, `\b{wb}`, and `\b{lb}` (and their `\B` forms) execute natively in Joni from reproducible pinned Perl 5.44 Unicode 17.0 data. The complete 239,866-assertion generated boundary corpus passes on both backends. - ✅ **Variable Interpolation in Regex**: Features like `${var}` for embedding variables. - ✅ **Non-capturing groups**: `(?:...)` is implemented. diff --git a/src/test/resources/unit/regex/vertical_whitespace_escape.t b/src/test/resources/unit/regex/vertical_whitespace_escape.t new file mode 100644 index 0000000000..09484757a1 --- /dev/null +++ b/src/test/resources/unit/regex/vertical_whitespace_escape.t @@ -0,0 +1,31 @@ +use strict; +use warnings; +use feature 'unicode_strings'; +use Test::More; + +my @vertical = (0x0A, 0x0B, 0x0C, 0x0D, 0x85, 0x2028, 0x2029); +my @other = (0x00, 0x09, 0x20, 0x56, 0xA0, 0x100); + +sub check_code_point { + my ($code, $expected, $upgraded) = @_; + my $character = chr($code); + utf8::upgrade($character) if $upgraded; + my $mode = $upgraded ? 'upgraded' : 'native'; + my $label = sprintf 'U+%04X %s', $code, $mode; + + is($character =~ /\v/ ? 1 : 0, $expected, "direct v $label"); + is($character =~ /\V/ ? 1 : 0, 1 - $expected, "direct V $label"); + is($character =~ /[\v]/ ? 1 : 0, $expected, "class v $label"); + is($character =~ /[\V]/ ? 1 : 0, 1 - $expected, "class V $label"); +} + +for my $code (@vertical) { + check_code_point($code, 1, 0); + check_code_point($code, 1, 1) if $code <= 0xFF; +} +for my $code (@other) { + check_code_point($code, 0, 0); + check_code_point($code, 0, 1) if $code <= 0xFF; +} + +done_testing(); diff --git a/third_party/joni/src/org/joni/Lexer.java b/third_party/joni/src/org/joni/Lexer.java index 6ca7d05289..99f180b79f 100644 --- a/third_party/joni/src/org/joni/Lexer.java +++ b/third_party/joni/src/org/joni/Lexer.java @@ -36,10 +36,16 @@ import org.joni.exception.ErrorMessages; class Lexer extends ScannerSupport { + private static final int[] PERL_VERTICAL_WHITESPACE_CODES = { + 0x0a, 0x0d, 0x85, 0x2028, 0x2029 + }; + protected final Regex regex; protected final ScanEnvironment env; protected final Syntax syntax; // fast access to syntax protected final Token token = new Token(); // current token + private int perlVerticalWhitespaceTokenIndex = -1; + private boolean perlVerticalWhitespaceNegated; protected Lexer(Regex regex, Syntax syntax, byte[]bytes, int p, int end, WarnCallback warnings) { super(regex.enc, bytes, p, end); @@ -546,6 +552,70 @@ private void fetchTokenInCCFor_charType(boolean flag, int type) { token.setPropNot(flag); } + private boolean usesPerlVerticalWhitespaceEscape() { + return syntax.op2EscVVerticalWhiteSpace() + || syntax.op2OptionPerl() && (syntax == Syntax.Perl + || syntax == Syntax.PerlNG || "PERLONJAVA".equals(syntax.name)); + } + + private void startPerlVerticalWhitespace(boolean negated, TokenType openType) { + perlVerticalWhitespaceTokenIndex = 0; + perlVerticalWhitespaceNegated = negated; + token.type = openType; + } + + private TokenType fetchPerlVerticalWhitespaceToken() { + token.base = 0; + token.escaped = false; + + int index = perlVerticalWhitespaceTokenIndex++; + if (perlVerticalWhitespaceNegated) { + if (index == 0) { + token.type = TokenType.CHAR; + token.setC('^'); + return token.type; + } + index--; + } + + switch (index) { + case 0: + token.type = TokenType.CODE_POINT; + token.setCode(PERL_VERTICAL_WHITESPACE_CODES[0]); + break; + case 1: + token.type = TokenType.CC_RANGE; + token.setC('-'); + break; + case 2: + token.type = TokenType.CODE_POINT; + token.setCode(PERL_VERTICAL_WHITESPACE_CODES[1]); + break; + case 3: + token.type = TokenType.CODE_POINT; + token.setCode(PERL_VERTICAL_WHITESPACE_CODES[2]); + break; + case 4: + token.type = TokenType.CODE_POINT; + token.setCode(PERL_VERTICAL_WHITESPACE_CODES[3]); + break; + case 5: + token.type = TokenType.CC_RANGE; + token.setC('-'); + break; + case 6: + token.type = TokenType.CODE_POINT; + token.setCode(PERL_VERTICAL_WHITESPACE_CODES[4]); + break; + default: + token.type = TokenType.CC_CLOSE; + token.setC(']'); + perlVerticalWhitespaceTokenIndex = -1; + break; + } + return token.type; + } + private void fetchTokenInCCFor_p() { int c2 = peek(); // !!! migrate to peekIs if (c2 == '{' && syntax.op2EscPBraceCharProperty()) { @@ -816,6 +886,9 @@ private void fetchTokenInCCFor_and() { } protected final TokenType fetchTokenInCC() { + if (perlVerticalWhitespaceTokenIndex >= 0) { + return fetchPerlVerticalWhitespaceToken(); + } if (!left()) { token.type = TokenType.EOT; return token.type; @@ -888,6 +961,14 @@ protected final TokenType fetchTokenInCC() { fetchTokenInCCFor_digit(); break; + case 'v': + case 'V': + if (usesPerlVerticalWhitespaceEscape()) { + startPerlVerticalWhitespace(c == 'V', TokenType.CC_CC_OPEN); + break; + } + // fall through + default: unfetch(); fetchEscapedValue(); @@ -1367,6 +1448,13 @@ protected final void fetchToken() { case 'K': if (syntax.op2EscCapitalKKeep()) token.type = TokenType.KEEP; break; + case 'v': + case 'V': + if (usesPerlVerticalWhitespaceEscape()) { + startPerlVerticalWhitespace(c == 'V', TokenType.CC_OPEN); + break; + } + // fall through default: unfetch(); fetchEscapedValue(); diff --git a/third_party/joni/test/org/joni/test/TestPerlVerticalWhitespace.java b/third_party/joni/test/org/joni/test/TestPerlVerticalWhitespace.java new file mode 100644 index 0000000000..1e59180a58 --- /dev/null +++ b/third_party/joni/test/org/joni/test/TestPerlVerticalWhitespace.java @@ -0,0 +1,90 @@ +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + * of the Software, and to permit persons to whom the Software is furnished to do + * so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +package org.joni.test; + +import static org.junit.Assert.assertEquals; + +import java.nio.charset.StandardCharsets; + +import org.jcodings.specific.UTF8Encoding; +import org.joni.Option; +import org.joni.Regex; +import org.joni.Syntax; +import org.junit.Test; + +public class TestPerlVerticalWhitespace { + private static final String VERTICAL = "\n\u000b\f\r\u0085\u2028\u2029"; + private static final String OTHER = "\0\t V\u00a0\u0100"; + + private static int search(String pattern, String input, Syntax syntax) { + byte[] patternBytes = pattern.getBytes(StandardCharsets.UTF_8); + byte[] inputBytes = input.getBytes(StandardCharsets.UTF_8); + Regex regex = new Regex(patternBytes, 0, patternBytes.length, Option.NONE, + UTF8Encoding.INSTANCE, syntax); + return regex.matcher(inputBytes).search(0, inputBytes.length, Option.NONE); + } + + private static void assertMatch(String pattern, String input, Syntax syntax) { + assertEquals(0, search("\\A(?:" + pattern + ")\\z", input, syntax)); + } + + private static void assertNoMatch(String pattern, String input, Syntax syntax) { + assertEquals(-1, search("\\A(?:" + pattern + ")\\z", input, syntax)); + } + + @Test + public void implementsPerlVerticalWhitespaceInsideAndOutsideClasses() { + for (int offset = 0; offset < VERTICAL.length();) { + int codePoint = VERTICAL.codePointAt(offset); + String character = new String(Character.toChars(codePoint)); + assertMatch("\\v", character, Syntax.PerlNG); + assertMatch("[\\v]", character, Syntax.PerlNG); + assertNoMatch("\\V", character, Syntax.PerlNG); + assertNoMatch("[\\V]", character, Syntax.PerlNG); + offset += Character.charCount(codePoint); + } + + for (int offset = 0; offset < OTHER.length();) { + int codePoint = OTHER.codePointAt(offset); + String character = new String(Character.toChars(codePoint)); + assertNoMatch("\\v", character, Syntax.PerlNG); + assertNoMatch("[\\v]", character, Syntax.PerlNG); + assertMatch("\\V", character, Syntax.PerlNG); + assertMatch("[\\V]", character, Syntax.PerlNG); + offset += Character.charCount(codePoint); + } + + assertMatch("\\v+", VERTICAL, Syntax.PerlNG); + assertMatch("[A\\v]", "A", Syntax.PerlNG); + assertMatch("[A\\V]", "Z", Syntax.PerlNG); + } + + @Test + public void preservesNonPerlVerticalTabEscapes() { + for (Syntax syntax : new Syntax[] {Syntax.RUBY, Syntax.Java, Syntax.ECMAScript}) { + assertMatch("\\v", "\u000b", syntax); + assertNoMatch("\\v", "\n", syntax); + assertMatch("\\V", "V", syntax); + assertNoMatch("\\V", "A", syntax); + assertMatch("[\\v]", "\u000b", syntax); + assertMatch("[\\V]", "V", syntax); + } + } +} From 540a108c73d3a61e6b6edb84401c4f59a8c72f47 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:50 +0200 Subject: [PATCH 30/60] feat(regex): add pinned Unicode decomposition types (#1035) Squashed integration of PR #1035 at 111fa207a. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 41 +- ...te_perl_unicode_decomposition_type_data.pl | 341 ++++++++++++++++ docs/reference/feature-matrix.md | 4 +- .../PerlUnicodeDecompositionTypeData.java | 368 ++++++++++++++++++ .../runtime/regex/UnicodeResolver.java | 40 +- .../PerlUnicodeDecompositionTypeDataTest.java | 192 +++++++++ .../regex/unicode_decomposition_type_data.t | 62 +++ 7 files changed, 1031 insertions(+), 17 deletions(-) create mode 100644 dev/tools/generate_perl_unicode_decomposition_type_data.pl create mode 100644 src/main/java/org/perlonjava/runtime/regex/PerlUnicodeDecompositionTypeData.java create mode 100644 src/test/java/org/perlonjava/runtime/regex/PerlUnicodeDecompositionTypeDataTest.java create mode 100644 src/test/resources/unit/regex/unicode_decomposition_type_data.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 5440afd580..0e1d2c96a3 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,7 +224,7 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1033, stacked +The published integration stack is preserved through draft PR #1034, stacked on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the @@ -236,9 +236,10 @@ progression and pinned Perl 5.44 Unicode 17.0 Age properties. PR #1030 adds binary `ASCII_Hex_Digit` values, pinned General_Category sets, and exact native line boundaries. PR #1031 adds pinned Canonical_Combining_Class sets and valid empty-property rendering. PR #1032 integrates native numeric escapes through -U+10FFFF; PR #1033 adds pinned Bidi_Class sets. The new WIP integrates native -vertical-whitespace escapes while Decomposition_Type, East_Asian_Width, and -Numeric_Value slices advance independently. +U+10FFFF; PR #1033 adds pinned Bidi_Class sets; PR #1034 integrates native +vertical-whitespace escapes. The current WIP integrates Decomposition_Type; +East_Asian_Width is ready for the next focused integration, while Numeric_Value +and Joining_Group data slices advance independently. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -328,6 +329,15 @@ without changing non-Perl Joni syntax behavior. The focused oracle passes from 2,052/2,560 to 2,560/2,560 on both execution backends with zero numbered regressions, closing its entire 508-assertion Joni gap. +`Decomposition_Type`/`dt` assignments now resolve all 18 atomic values plus +Perl's composite `Non_Canonical` value from a complete pinned Unicode 17 +partition. Short/long and loose aliases, ordered `None` defaults, the exact +case-sensitive `Is` assignment prefix, and invalid-value rejection are covered. +The focused oracle passes 45/45 on system Perl, JVM, and interpreter. Chunk 01 +gains 640 assertions to 34,892/41,843 identically on both execution backends +with zero numbered regressions, raising current generated evidence to +342,656/407,367. + Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while ordinary and substitution callbacks execute, without misclassifying escaped or @@ -578,6 +588,11 @@ is retained for now. partition with loose aliases and ordered missing defaults. The focused oracle passes 99/99 and chunk 01 gains 736 assertions to 34,252/41,843 on both backends with zero numbered regressions. + - [x] Generated and integrated a complete pinned Unicode 17.0 + Decomposition_Type partition, including Perl's composite `Non_Canonical` + value and exact `Is` prefix policy. The focused oracle passes 45/45 and + chunk 01 gains 640 assertions to 34,892/41,843 on both backends with zero + numbered regressions. - [x] Integrated native Perl `\v`/`\V` dispatch inside and outside character classes (`1eff1db97`, integrated as `6328935cd`). The focused oracle passes 92/92 and unchanged `reg_posixcc.t` passes 2,560/2,560 on both backends. @@ -623,20 +638,18 @@ is retained for now. ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1033. Publish the validated - native vertical-whitespace slice, then integrate the completed - Decomposition_Type data handoff in a separate focused WIP PR. +1. Land review-ready PR #1024 and draft PRs #1025–#1034. Publish the validated + Decomposition_Type integration in a separate focused WIP PR, then integrate + East_Asian_Width without squashing the independently reviewed data commit. 2. Preserve the now-complete native Joni boundary corpus at 239,866/239,866: sentence chunk 05, line chunks 06–09, and word chunk 10 must remain exact on JVM and interpreter while property and parser work continues. -3. Integrate the independently generated Decomposition_Type, - East_Asian_Width, and Numeric_Value slices, then continue the measured - residual order: binary - property values, Block, Script/Script_Extensions, Numeric_Value, - Joining_Group, and break-property values. Preserve pinned Perl 5.44 +3. Integrate the independently generated East_Asian_Width, Numeric_Value, and + Joining_Group slices, then continue the measured residual order: binary + property values, Block, Script/Script_Extensions, and break-property values. + Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. - Integrate native `\v`/`\V` after the numeric Lexer slice and require exact - `reg_posixcc.t` parity. + Keep native `\v`/`\V` exact at 2,560/2,560 in `reg_posixcc.t`. 4. Rerun generated property chunks 01–04 on both backends with the classified 600-second bound and retain complete TAP/JSON. After the two fatal roots and native line boundaries integrate, refresh the complete forced-Joni 80-file diff --git a/dev/tools/generate_perl_unicode_decomposition_type_data.pl b/dev/tools/generate_perl_unicode_decomposition_type_data.pl new file mode 100644 index 0000000000..1088707b98 --- /dev/null +++ b/dev/tools/generate_perl_unicode_decomposition_type_data.pl @@ -0,0 +1,341 @@ +#!/usr/bin/env perl +use strict; +use warnings; +use Digest::SHA qw(sha256_hex); +use File::Spec; +use FindBin; + +binmode STDOUT, ':raw'; + +my $expected_version = '17.0.0'; +my $unicore = File::Spec->catdir($FindBin::Bin, '..', '..', 'perl5', 'lib', 'unicore'); +my %sources = ( + Version => [File::Spec->catfile($unicore, 'version'), + '8c30575264b2772c7a69c5bb6069a28f0e0a7a0df735871bde2d99ee674316ac'], + Decomposition_Type => [File::Spec->catfile($unicore, 'extracted', 'DDecompositionType.txt'), + 'f44e5ceaf40edc1fe06ea0404e8bebc7d356dcc38aac076543b6874008a06e3e'], + Property_Aliases => [File::Spec->catfile($unicore, 'PropertyAliases.txt'), + '4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb'], + Property_Value_Aliases => [File::Spec->catfile($unicore, 'PropValueAliases.txt'), + '670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01'], +); + +sub source_text { + my ($name) = @_; + my ($path, $expected_hash) = @{$sources{$name}}; + open my $input, '<:raw', $path or die "Cannot read $path: $!\n"; + local $/; + my $text = <$input>; + close $input or die "Cannot close $path: $!\n"; + my $actual_hash = sha256_hex($text); + die "$path SHA-256 mismatch: expected $expected_hash, found $actual_hash\n" + unless $actual_hash eq $expected_hash; + return ($path, $text); +} + +sub trim { + my ($text) = @_; + $text =~ s/^\s+|\s+$//g; + return $text; +} + +sub loose { + my ($text) = @_; + $text = lc $text; + $text =~ s/[\s_-]+//g; + return $text; +} + +sub range_from_text { + my ($range) = @_; + my ($first, $last) = split /\.\./, $range; + return (hex($first), hex(defined $last ? $last : $first)); +} + +sub verify_unicode_notice { + my ($path, $text) = @_; + die "$path does not preserve the Unicode copyright notice\n" + unless $text =~ /^# © 2025 Unicode®, Inc\.$/m; + die "$path does not preserve the Unicode terms notice\n" + unless $text =~ m{^# For terms of use and license, see https://www\.unicode\.org/terms_of_use\.html$}m; +} + +my ($version_path, $version_text) = source_text('Version'); +$version_text =~ s/\s+\z//; +die "Expected Unicode $expected_version, found '$version_text' in $version_path\n" + unless $version_text eq $expected_version; + +my ($data_path, $data_text) = source_text('Decomposition_Type'); +die "$data_path is not pinned Unicode $expected_version data\n" + unless $data_text =~ /^# DerivedDecompositionType-\Q$expected_version\E\.txt$/m; +verify_unicode_notice($data_path, $data_text); + +my (@ranges, @missing); +for my $line (split /\n/, $data_text) { + if ($line =~ /^#\s*\@missing:\s*([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z0-9_]+)/) { + my ($first, $last) = range_from_text($1); + push @missing, [$first, $last, $2]; + next; + } + next unless $line =~ /^([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z0-9_]+)/; + my ($first, $last) = range_from_text($1); + push @ranges, [$first, $last, $2]; +} +die "No Decomposition_Type ranges found in $data_path\n" unless @ranges; +die "No ordered \@missing rules found in $data_path\n" unless @missing; +@ranges = sort { $a->[0] <=> $b->[0] } @ranges; +for my $index (1 .. $#ranges) { + die sprintf("Overlapping Decomposition_Type ranges at U+%04X\n", $ranges[$index][0]) + if $ranges[$index][0] <= $ranges[$index - 1][1]; +} + +my ($property_path, $property_text) = source_text('Property_Aliases'); +die "$property_path is not pinned Unicode $expected_version data\n" + unless $property_text =~ /^# PropertyAliases-\Q$expected_version\E\.txt$/m; +verify_unicode_notice($property_path, $property_text); +my @property_aliases; +for my $line (split /\n/, $property_text) { + $line =~ s/#.*//; + my @fields = map { trim($_) } split /;/, $line, -1; + next unless @fields >= 2 && $fields[0] eq 'dt'; + @property_aliases = grep { length } @fields; +} +die "Missing dt aliases in $property_path\n" unless @property_aliases; + +my ($value_path, $value_text) = source_text('Property_Value_Aliases'); +die "$value_path is not pinned Unicode $expected_version data\n" + unless $value_text =~ /^# PropertyValueAliases-\Q$expected_version\E\.txt$/m; +verify_unicode_notice($value_path, $value_text); +my (@values, %value_index, %value_aliases); +for my $line (split /\n/, $value_text) { + $line =~ s/#.*//; + my @fields = map { trim($_) } split /;/, $line, -1; + next unless @fields >= 3 && $fields[0] eq 'dt'; + my ($short, $canonical, @extra) = @fields[1 .. $#fields]; + if (!exists $value_index{$canonical}) { + $value_index{$canonical} = scalar @values; + push @values, $canonical; + } + my $id = $value_index{$canonical}; + for my $alias (grep { length } ($short, $canonical, @extra)) { + my $normalized = loose($alias); + die "Conflicting Decomposition_Type alias '$alias'\n" + if exists $value_aliases{$normalized} && $value_aliases{$normalized} != $id; + $value_aliases{$normalized} = $id; + } +} +die "Expected 18 Decomposition_Type values, found " . scalar(@values) . "\n" + unless @values == 18; + +# Perl adds one valid union value beyond the Unicode property-value aliases. +# perl5/lib/unicore/mktables names it Non_Canon / Non_Canonical and defines it +# as the union of every non-canonical decomposition type. +my $non_canonical_id = scalar @values; +push @values, 'Non_Canonical'; +$value_index{Non_Canonical} = $non_canonical_id; +$value_aliases{loose('Non_Canon')} = $non_canonical_id; +$value_aliases{loose('Non_Canonical')} = $non_canonical_id; + +for my $range (@ranges, @missing) { + die "Unknown Decomposition_Type value '$range->[2]'\n" + unless exists $value_index{$range->[2]}; +} + +my %starts = (0 => 1); +for my $range (@ranges, @missing) { + die "Invalid Unicode range U+" . sprintf('%X', $range->[0]) . "..U+" + . sprintf('%X', $range->[1]) . "\n" + if $range->[0] < 0 || $range->[1] > 0x10ffff || $range->[0] > $range->[1]; + $starts{$range->[0]} = 1; + $starts{$range->[1] + 1} = 1 if $range->[1] < 0x10ffff; +} +my @starts = sort { $a <=> $b } keys %starts; + +sub value_at { + my ($code, $explicit, $defaults) = @_; + my $value; + # UAX #44 applies @missing rules in source order; a later matching rule wins. + for my $range (@$defaults) { + $value = $range->[2] if $range->[0] <= $code && $code <= $range->[1]; + } + my ($low, $high) = (0, $#$explicit); + while ($low <= $high) { + my $middle = ($low + $high) >> 1; + if ($explicit->[$middle][0] <= $code) { + $low = $middle + 1; + } else { + $high = $middle - 1; + } + } + $value = $explicit->[$high][2] + if $high >= 0 && $code <= $explicit->[$high][1]; + die sprintf("No Decomposition_Type value covers U+%04X\n", $code) + unless defined $value; + return $value; +} + +# Guard the ordering contract even though Unicode 17 currently has one rule. +my @ordered_missing_probe = ( + [0x0000, 0x10ffff, 'None'], + [0x1000, 0x10ff, 'Compat'], + [0x1080, 0x10af, 'Wide'], +); +my @explicit_probe = ([0x10a0, 0x10a0, 'Canonical']); +my @probe_codes = (0x0fff, 0x1000, 0x107f, 0x1080, 0x109f, 0x10a0, 0x10af, 0x10b0, 0x1100); +my @probe_expected = qw(None Compat Compat Wide Wide Canonical Wide Compat None); +for my $index (0 .. $#probe_codes) { + die "Ordered \@missing rule self-check failed\n" + unless value_at($probe_codes[$index], \@explicit_probe, \@ordered_missing_probe) + eq $probe_expected[$index]; +} + +my (@coalesced_starts, @coalesced_values); +for my $code (@starts) { + my $id = $value_index{value_at($code, \@ranges, \@missing)}; + next if @coalesced_values && $id == $coalesced_values[-1]; + push @coalesced_starts, $code; + push @coalesced_values, $id; +} +die "Generated partition does not begin at U+0000\n" + unless @coalesced_starts && $coalesced_starts[0] == 0; + +my %constant_for; +for my $value (@values) { + my $constant = uc $value; + $constant =~ s/[^A-Z0-9]+/_/g; + $constant_for{$value} = $constant; +} + +print <<'HEADER'; +package org.perlonjava.runtime.regex; + +/* + * Generated from Perl 5.44's pinned Unicode Character Database by + * dev/tools/generate_perl_unicode_decomposition_type_data.pl. Do not edit manually. + * + * Unicode data source copyright: + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in + * the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + */ +final class PerlUnicodeDecompositionTypeData { +HEADER + +print " static final String UNICODE_VERSION = \"$expected_version\";\n"; +for my $value (@values) { + print " static final byte $constant_for{$value} = $value_index{$value};\n"; +} +print " static final byte INVALID = -1;\n\n"; +print " private static final String[] CANONICAL_NAMES = {\n "; +print join(', ', map { qq{"$_"} } @values); +print "\n };\n\n"; +print " private static final int[] STARTS = {\n"; +for (my $i = 0; $i < @coalesced_starts; $i += 10) { + my $end = $i + 9 < $#coalesced_starts ? $i + 9 : $#coalesced_starts; + print " ", join(', ', map { sprintf '0x%X', $coalesced_starts[$_] } $i .. $end), ",\n"; +} +print " };\n\n private static final byte[] VALUES = {\n"; +for (my $i = 0; $i < @coalesced_values; $i += 18) { + my $end = $i + 17 < $#coalesced_values ? $i + 17 : $#coalesced_values; + print " ", join(', ', @coalesced_values[$i .. $end]), ",\n"; +} +print <<'METHODS'; + }; + + static byte propertyOf(int codePoint) { + if (codePoint < 0 || codePoint > 0x10ffff) { + throw new IllegalArgumentException("Not a Unicode code point: " + codePoint); + } + int low = 0; + int high = STARTS.length - 1; + while (low <= high) { + int middle = (low + high) >>> 1; + if (STARTS[middle] <= codePoint) { + low = middle + 1; + } else { + high = middle - 1; + } + } + return VALUES[high]; + } + + static int rangeCount() { + return STARTS.length; + } + + static int rangeStart(int index) { + return STARTS[index]; + } + + static int rangeEnd(int index) { + return index + 1 < STARTS.length ? STARTS[index + 1] - 1 : 0x10ffff; + } + + static byte rangeValue(int index) { + return VALUES[index]; + } + + static boolean isPropertyAlias(String alias) { + boolean hasIsPrefix = alias != null && alias.startsWith("Is"); + String normalized = loose(hasIsPrefix ? alias.substring(2) : alias); + switch (normalized) { +METHODS +for my $alias (sort { loose($a) cmp loose($b) } @property_aliases) { + print ' case "', loose($alias), "\":\n"; +} +print <<'PROPERTY_FOOTER'; + return true; + default: + return false; + } + } + + static byte valueForAlias(String alias) { + String normalized = loose(alias); + switch (normalized) { +PROPERTY_FOOTER +my %aliases_by_id; +for my $alias (sort keys %value_aliases) { + push @{$aliases_by_id{$value_aliases{$alias}}}, $alias; +} +for my $id (0 .. $#values) { + for my $alias (@{$aliases_by_id{$id}}) { + print " case \"$alias\":\n"; + } + print " return $constant_for{$values[$id]};\n"; +} +print <<'FOOTER'; + default: + return INVALID; + } + } + + static String canonicalValueName(byte value) { + return value >= 0 && value < CANONICAL_NAMES.length + ? CANONICAL_NAMES[value] : null; + } + + static boolean matches(byte property, byte requested) { + if (requested == NON_CANONICAL) { + return property != CANONICAL && property != NONE; + } + return property == requested; + } + + private static String loose(String alias) { + if (alias == null) return ""; + StringBuilder normalized = new StringBuilder(alias.length()); + for (int i = 0; i < alias.length(); i++) { + char character = alias.charAt(i); + if (character == '_' || character == '-' || character == ' ' + || (character >= '\t' && character <= '\r')) continue; + normalized.append(Character.toLowerCase(character)); + } + return normalized.toString(); + } + + private PerlUnicodeDecompositionTypeData() { + } +} +FOOTER diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index 8ca13fc07e..f9a8158547 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -379,7 +379,7 @@ my @copy = @{$z}; # ERROR - ✅ **Backreferences to Named Groups**: Using `\k` or `\g{name}` for backreferences to named groups is supported. - ✅ **Relative Backreferences**: Using `\g{-n}` for relative backreferences. - ✅ **Basic Unicode Properties**: Common `\p{...}` and `\P{...}` forms such as `\p{L}` execute through Joni. -- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, Canonical_Combining_Class, and Bidi_Class assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, and reserved empty values; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. +- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, and Decomposition_Type assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, and reserved or composite values; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. - ✅ **Possessive Quantifiers**: Quantifiers like `*+`, `++`, `?+`, and `{n,m}+`, which disable backtracking, are supported. - ✅ **Atomic Grouping**: Use of `(?>...)` for atomic groups is supported. - ✅ **`\K` assertion**: Keep left — in `s///`, text before `\K` is preserved; match variables reflect only the portion after `\K`. @@ -398,7 +398,7 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 342,016/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 342,656/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. - ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. - ✅ **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter. - ✅ **Unicode Word Boundaries**: Native `\b{wb}`/`\B{wb}` implement WB1–WB16 and WB999 from reproducible Perl 5.44 Unicode 17.0 word-break and extended-pictographic tables. Authoritative chunk 10 passes 19,510/19,510 identically on JVM and interpreter. diff --git a/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeDecompositionTypeData.java b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeDecompositionTypeData.java new file mode 100644 index 0000000000..dfe3bd02f9 --- /dev/null +++ b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeDecompositionTypeData.java @@ -0,0 +1,368 @@ +package org.perlonjava.runtime.regex; + +/* + * Generated from Perl 5.44's pinned Unicode Character Database by + * dev/tools/generate_perl_unicode_decomposition_type_data.pl. Do not edit manually. + * + * Unicode data source copyright: + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in + * the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + */ +final class PerlUnicodeDecompositionTypeData { + static final String UNICODE_VERSION = "17.0.0"; + static final byte CANONICAL = 0; + static final byte COMPAT = 1; + static final byte CIRCLE = 2; + static final byte FINAL = 3; + static final byte FONT = 4; + static final byte FRACTION = 5; + static final byte INITIAL = 6; + static final byte ISOLATED = 7; + static final byte MEDIAL = 8; + static final byte NARROW = 9; + static final byte NOBREAK = 10; + static final byte NONE = 11; + static final byte SMALL = 12; + static final byte SQUARE = 13; + static final byte SUB = 14; + static final byte SUPER = 15; + static final byte VERTICAL = 16; + static final byte WIDE = 17; + static final byte NON_CANONICAL = 18; + static final byte INVALID = -1; + + private static final String[] CANONICAL_NAMES = { + "Canonical", "Compat", "Circle", "Final", "Font", "Fraction", "Initial", "Isolated", "Medial", "Narrow", "Nobreak", "None", "Small", "Square", "Sub", "Super", "Vertical", "Wide", "Non_Canonical" + }; + + private static final int[] STARTS = { + 0x0, 0xA0, 0xA1, 0xA8, 0xA9, 0xAA, 0xAB, 0xAF, 0xB0, 0xB2, + 0xB4, 0xB6, 0xB8, 0xB9, 0xBB, 0xBC, 0xBF, 0xC0, 0xC6, 0xC7, + 0xD0, 0xD1, 0xD7, 0xD9, 0xDE, 0xE0, 0xE6, 0xE7, 0xF0, 0xF1, + 0xF7, 0xF9, 0xFE, 0xFF, 0x110, 0x112, 0x126, 0x128, 0x131, 0x132, + 0x134, 0x138, 0x139, 0x13F, 0x141, 0x143, 0x149, 0x14A, 0x14C, 0x152, + 0x154, 0x166, 0x168, 0x17F, 0x180, 0x1A0, 0x1A2, 0x1AF, 0x1B1, 0x1C4, + 0x1CD, 0x1DD, 0x1DE, 0x1E4, 0x1E6, 0x1F1, 0x1F4, 0x1F6, 0x1F8, 0x21C, + 0x21E, 0x220, 0x226, 0x234, 0x2B0, 0x2B9, 0x2D8, 0x2DE, 0x2E0, 0x2E5, + 0x340, 0x342, 0x343, 0x345, 0x374, 0x375, 0x37A, 0x37B, 0x37E, 0x37F, + 0x384, 0x385, 0x38B, 0x38C, 0x38D, 0x38E, 0x391, 0x3AA, 0x3B1, 0x3CA, + 0x3CF, 0x3D0, 0x3D3, 0x3D5, 0x3D7, 0x3F0, 0x3F3, 0x3F4, 0x3F6, 0x3F9, + 0x3FA, 0x400, 0x402, 0x403, 0x404, 0x407, 0x408, 0x40C, 0x40F, 0x419, + 0x41A, 0x439, 0x43A, 0x450, 0x452, 0x453, 0x454, 0x457, 0x458, 0x45C, + 0x45F, 0x476, 0x478, 0x4C1, 0x4C3, 0x4D0, 0x4D4, 0x4D6, 0x4D8, 0x4DA, + 0x4E0, 0x4E2, 0x4E8, 0x4EA, 0x4F6, 0x4F8, 0x4FA, 0x587, 0x588, 0x622, + 0x627, 0x675, 0x679, 0x6C0, 0x6C1, 0x6C2, 0x6C3, 0x6D3, 0x6D4, 0x929, + 0x92A, 0x931, 0x932, 0x934, 0x935, 0x958, 0x960, 0x9CB, 0x9CD, 0x9DC, + 0x9DE, 0x9DF, 0x9E0, 0xA33, 0xA34, 0xA36, 0xA37, 0xA59, 0xA5C, 0xA5E, + 0xA5F, 0xB48, 0xB49, 0xB4B, 0xB4D, 0xB5C, 0xB5E, 0xB94, 0xB95, 0xBCA, + 0xBCD, 0xC48, 0xC49, 0xCC0, 0xCC1, 0xCC7, 0xCC9, 0xCCA, 0xCCC, 0xD4A, + 0xD4D, 0xDDA, 0xDDB, 0xDDC, 0xDDF, 0xE33, 0xE34, 0xEB3, 0xEB4, 0xEDC, + 0xEDE, 0xF0C, 0xF0D, 0xF43, 0xF44, 0xF4D, 0xF4E, 0xF52, 0xF53, 0xF57, + 0xF58, 0xF5C, 0xF5D, 0xF69, 0xF6A, 0xF73, 0xF74, 0xF75, 0xF77, 0xF78, + 0xF79, 0xF7A, 0xF81, 0xF82, 0xF93, 0xF94, 0xF9D, 0xF9E, 0xFA2, 0xFA3, + 0xFA7, 0xFA8, 0xFAC, 0xFAD, 0xFB9, 0xFBA, 0x1026, 0x1027, 0x10FC, 0x10FD, + 0x1B06, 0x1B07, 0x1B08, 0x1B09, 0x1B0A, 0x1B0B, 0x1B0C, 0x1B0D, 0x1B0E, 0x1B0F, + 0x1B12, 0x1B13, 0x1B3B, 0x1B3C, 0x1B3D, 0x1B3E, 0x1B40, 0x1B42, 0x1B43, 0x1B44, + 0x1D2C, 0x1D2F, 0x1D30, 0x1D3B, 0x1D3C, 0x1D4E, 0x1D4F, 0x1D62, 0x1D6B, 0x1D78, + 0x1D79, 0x1D9B, 0x1DC0, 0x1E00, 0x1E9A, 0x1E9B, 0x1E9C, 0x1EA0, 0x1EFA, 0x1F00, + 0x1F16, 0x1F18, 0x1F1E, 0x1F20, 0x1F46, 0x1F48, 0x1F4E, 0x1F50, 0x1F58, 0x1F59, + 0x1F5A, 0x1F5B, 0x1F5C, 0x1F5D, 0x1F5E, 0x1F5F, 0x1F7E, 0x1F80, 0x1FB5, 0x1FB6, + 0x1FBD, 0x1FBE, 0x1FBF, 0x1FC1, 0x1FC5, 0x1FC6, 0x1FD4, 0x1FD6, 0x1FDC, 0x1FDD, + 0x1FF0, 0x1FF2, 0x1FF5, 0x1FF6, 0x1FFE, 0x1FFF, 0x2000, 0x2002, 0x2007, 0x2008, + 0x200B, 0x2011, 0x2012, 0x2017, 0x2018, 0x2024, 0x2027, 0x202F, 0x2030, 0x2033, + 0x2035, 0x2036, 0x2038, 0x203C, 0x203D, 0x203E, 0x203F, 0x2047, 0x204A, 0x2057, + 0x2058, 0x205F, 0x2060, 0x2070, 0x2072, 0x2074, 0x2080, 0x208F, 0x2090, 0x209D, + 0x20A8, 0x20A9, 0x2100, 0x2102, 0x2103, 0x2104, 0x2105, 0x2108, 0x2109, 0x210A, + 0x2114, 0x2115, 0x2116, 0x2117, 0x2119, 0x211E, 0x2120, 0x2121, 0x2122, 0x2123, + 0x2124, 0x2125, 0x2126, 0x2127, 0x2128, 0x2129, 0x212A, 0x212C, 0x212E, 0x212F, + 0x2132, 0x2133, 0x2135, 0x2139, 0x213A, 0x213B, 0x213C, 0x2141, 0x2145, 0x214A, + 0x2150, 0x2160, 0x2180, 0x2189, 0x218A, 0x219A, 0x219C, 0x21AE, 0x21AF, 0x21CD, + 0x21D0, 0x2204, 0x2205, 0x2209, 0x220A, 0x220C, 0x220D, 0x2224, 0x2225, 0x2226, + 0x2227, 0x222C, 0x222E, 0x222F, 0x2231, 0x2241, 0x2242, 0x2244, 0x2245, 0x2247, + 0x2248, 0x2249, 0x224A, 0x2260, 0x2261, 0x2262, 0x2263, 0x226D, 0x2272, 0x2274, + 0x2276, 0x2278, 0x227A, 0x2280, 0x2282, 0x2284, 0x2286, 0x2288, 0x228A, 0x22AC, + 0x22B0, 0x22E0, 0x22E4, 0x22EA, 0x22EE, 0x2329, 0x232B, 0x2460, 0x2474, 0x24B6, + 0x24EB, 0x2A0C, 0x2A0D, 0x2A74, 0x2A77, 0x2ADC, 0x2ADD, 0x2C7C, 0x2C7D, 0x2C7E, + 0x2D6F, 0x2D70, 0x2E9F, 0x2EA0, 0x2EF3, 0x2EF4, 0x2F00, 0x2FD6, 0x3000, 0x3001, + 0x3036, 0x3037, 0x3038, 0x303B, 0x304C, 0x304D, 0x304E, 0x304F, 0x3050, 0x3051, + 0x3052, 0x3053, 0x3054, 0x3055, 0x3056, 0x3057, 0x3058, 0x3059, 0x305A, 0x305B, + 0x305C, 0x305D, 0x305E, 0x305F, 0x3060, 0x3061, 0x3062, 0x3063, 0x3065, 0x3066, + 0x3067, 0x3068, 0x3069, 0x306A, 0x3070, 0x3072, 0x3073, 0x3075, 0x3076, 0x3078, + 0x3079, 0x307B, 0x307C, 0x307E, 0x3094, 0x3095, 0x309B, 0x309D, 0x309E, 0x309F, + 0x30A0, 0x30AC, 0x30AD, 0x30AE, 0x30AF, 0x30B0, 0x30B1, 0x30B2, 0x30B3, 0x30B4, + 0x30B5, 0x30B6, 0x30B7, 0x30B8, 0x30B9, 0x30BA, 0x30BB, 0x30BC, 0x30BD, 0x30BE, + 0x30BF, 0x30C0, 0x30C1, 0x30C2, 0x30C3, 0x30C5, 0x30C6, 0x30C7, 0x30C8, 0x30C9, + 0x30CA, 0x30D0, 0x30D2, 0x30D3, 0x30D5, 0x30D6, 0x30D8, 0x30D9, 0x30DB, 0x30DC, + 0x30DE, 0x30F4, 0x30F5, 0x30F7, 0x30FB, 0x30FE, 0x30FF, 0x3100, 0x3131, 0x318F, + 0x3192, 0x31A0, 0x3200, 0x321F, 0x3220, 0x3244, 0x3248, 0x3250, 0x3251, 0x327F, + 0x3280, 0x32C0, 0x32CC, 0x32D0, 0x32FF, 0x3358, 0x3371, 0x33E0, 0x33FF, 0x3400, + 0xA69C, 0xA69E, 0xA770, 0xA771, 0xA7F1, 0xA7F5, 0xA7F8, 0xA7FA, 0xAB5C, 0xAB60, + 0xAB69, 0xAB6A, 0xAC00, 0xD7A4, 0xF900, 0xFA0E, 0xFA10, 0xFA11, 0xFA12, 0xFA13, + 0xFA15, 0xFA1F, 0xFA20, 0xFA21, 0xFA22, 0xFA23, 0xFA25, 0xFA27, 0xFA2A, 0xFA6E, + 0xFA70, 0xFADA, 0xFB00, 0xFB07, 0xFB13, 0xFB18, 0xFB1D, 0xFB1E, 0xFB1F, 0xFB20, + 0xFB2A, 0xFB37, 0xFB38, 0xFB3D, 0xFB3E, 0xFB3F, 0xFB40, 0xFB42, 0xFB43, 0xFB45, + 0xFB46, 0xFB4F, 0xFB50, 0xFB51, 0xFB52, 0xFB53, 0xFB54, 0xFB55, 0xFB56, 0xFB57, + 0xFB58, 0xFB59, 0xFB5A, 0xFB5B, 0xFB5C, 0xFB5D, 0xFB5E, 0xFB5F, 0xFB60, 0xFB61, + 0xFB62, 0xFB63, 0xFB64, 0xFB65, 0xFB66, 0xFB67, 0xFB68, 0xFB69, 0xFB6A, 0xFB6B, + 0xFB6C, 0xFB6D, 0xFB6E, 0xFB6F, 0xFB70, 0xFB71, 0xFB72, 0xFB73, 0xFB74, 0xFB75, + 0xFB76, 0xFB77, 0xFB78, 0xFB79, 0xFB7A, 0xFB7B, 0xFB7C, 0xFB7D, 0xFB7E, 0xFB7F, + 0xFB80, 0xFB81, 0xFB82, 0xFB83, 0xFB84, 0xFB85, 0xFB86, 0xFB87, 0xFB88, 0xFB89, + 0xFB8A, 0xFB8B, 0xFB8C, 0xFB8D, 0xFB8E, 0xFB8F, 0xFB90, 0xFB91, 0xFB92, 0xFB93, + 0xFB94, 0xFB95, 0xFB96, 0xFB97, 0xFB98, 0xFB99, 0xFB9A, 0xFB9B, 0xFB9C, 0xFB9D, + 0xFB9E, 0xFB9F, 0xFBA0, 0xFBA1, 0xFBA2, 0xFBA3, 0xFBA4, 0xFBA5, 0xFBA6, 0xFBA7, + 0xFBA8, 0xFBA9, 0xFBAA, 0xFBAB, 0xFBAC, 0xFBAD, 0xFBAE, 0xFBAF, 0xFBB0, 0xFBB1, + 0xFBB2, 0xFBD3, 0xFBD4, 0xFBD5, 0xFBD6, 0xFBD7, 0xFBD8, 0xFBD9, 0xFBDA, 0xFBDB, + 0xFBDC, 0xFBDD, 0xFBDF, 0xFBE0, 0xFBE1, 0xFBE2, 0xFBE3, 0xFBE4, 0xFBE5, 0xFBE6, + 0xFBE7, 0xFBE8, 0xFBE9, 0xFBEA, 0xFBEB, 0xFBEC, 0xFBED, 0xFBEE, 0xFBEF, 0xFBF0, + 0xFBF1, 0xFBF2, 0xFBF3, 0xFBF4, 0xFBF5, 0xFBF6, 0xFBF7, 0xFBF8, 0xFBF9, 0xFBFA, + 0xFBFB, 0xFBFC, 0xFBFD, 0xFBFE, 0xFBFF, 0xFC00, 0xFC64, 0xFC97, 0xFCDF, 0xFCF5, + 0xFD11, 0xFD2D, 0xFD34, 0xFD3C, 0xFD3D, 0xFD3E, 0xFD50, 0xFD51, 0xFD52, 0xFD58, + 0xFD59, 0xFD5A, 0xFD5C, 0xFD5E, 0xFD60, 0xFD62, 0xFD63, 0xFD64, 0xFD65, 0xFD66, + 0xFD68, 0xFD69, 0xFD6B, 0xFD6C, 0xFD6D, 0xFD6E, 0xFD70, 0xFD71, 0xFD72, 0xFD74, + 0xFD77, 0xFD78, 0xFD7D, 0xFD7E, 0xFD83, 0xFD84, 0xFD86, 0xFD87, 0xFD88, 0xFD8B, + 0xFD8C, 0xFD90, 0xFD92, 0xFD96, 0xFD98, 0xFD99, 0xFD9D, 0xFD9E, 0xFDB4, 0xFDB6, + 0xFDB8, 0xFDB9, 0xFDBA, 0xFDBB, 0xFDC3, 0xFDC6, 0xFDC8, 0xFDF0, 0xFDFD, 0xFE10, + 0xFE1A, 0xFE30, 0xFE45, 0xFE47, 0xFE49, 0xFE50, 0xFE53, 0xFE54, 0xFE67, 0xFE68, + 0xFE6C, 0xFE70, 0xFE71, 0xFE72, 0xFE73, 0xFE74, 0xFE75, 0xFE76, 0xFE77, 0xFE78, + 0xFE79, 0xFE7A, 0xFE7B, 0xFE7C, 0xFE7D, 0xFE7E, 0xFE7F, 0xFE80, 0xFE82, 0xFE83, + 0xFE84, 0xFE85, 0xFE86, 0xFE87, 0xFE88, 0xFE89, 0xFE8A, 0xFE8B, 0xFE8C, 0xFE8D, + 0xFE8E, 0xFE8F, 0xFE90, 0xFE91, 0xFE92, 0xFE93, 0xFE94, 0xFE95, 0xFE96, 0xFE97, + 0xFE98, 0xFE99, 0xFE9A, 0xFE9B, 0xFE9C, 0xFE9D, 0xFE9E, 0xFE9F, 0xFEA0, 0xFEA1, + 0xFEA2, 0xFEA3, 0xFEA4, 0xFEA5, 0xFEA6, 0xFEA7, 0xFEA8, 0xFEA9, 0xFEAA, 0xFEAB, + 0xFEAC, 0xFEAD, 0xFEAE, 0xFEAF, 0xFEB0, 0xFEB1, 0xFEB2, 0xFEB3, 0xFEB4, 0xFEB5, + 0xFEB6, 0xFEB7, 0xFEB8, 0xFEB9, 0xFEBA, 0xFEBB, 0xFEBC, 0xFEBD, 0xFEBE, 0xFEBF, + 0xFEC0, 0xFEC1, 0xFEC2, 0xFEC3, 0xFEC4, 0xFEC5, 0xFEC6, 0xFEC7, 0xFEC8, 0xFEC9, + 0xFECA, 0xFECB, 0xFECC, 0xFECD, 0xFECE, 0xFECF, 0xFED0, 0xFED1, 0xFED2, 0xFED3, + 0xFED4, 0xFED5, 0xFED6, 0xFED7, 0xFED8, 0xFED9, 0xFEDA, 0xFEDB, 0xFEDC, 0xFEDD, + 0xFEDE, 0xFEDF, 0xFEE0, 0xFEE1, 0xFEE2, 0xFEE3, 0xFEE4, 0xFEE5, 0xFEE6, 0xFEE7, + 0xFEE8, 0xFEE9, 0xFEEA, 0xFEEB, 0xFEEC, 0xFEED, 0xFEEE, 0xFEEF, 0xFEF0, 0xFEF1, + 0xFEF2, 0xFEF3, 0xFEF4, 0xFEF5, 0xFEF6, 0xFEF7, 0xFEF8, 0xFEF9, 0xFEFA, 0xFEFB, + 0xFEFC, 0xFEFD, 0xFF01, 0xFF61, 0xFFBF, 0xFFC2, 0xFFC8, 0xFFCA, 0xFFD0, 0xFFD2, + 0xFFD8, 0xFFDA, 0xFFDD, 0xFFE0, 0xFFE7, 0xFFE8, 0xFFEF, 0x105C9, 0x105CA, 0x105E4, + 0x105E5, 0x10781, 0x10786, 0x10787, 0x107B1, 0x107B2, 0x107BB, 0x1109A, 0x1109B, 0x1109C, + 0x1109D, 0x110AB, 0x110AC, 0x1112E, 0x11130, 0x1134B, 0x1134D, 0x11383, 0x11384, 0x11385, + 0x11386, 0x1138E, 0x1138F, 0x11391, 0x11392, 0x113C5, 0x113C6, 0x113C7, 0x113C9, 0x114BB, + 0x114BD, 0x114BE, 0x114BF, 0x115BA, 0x115BC, 0x11938, 0x11939, 0x16121, 0x16129, 0x16D68, + 0x16D6B, 0x1CCD6, 0x1CCFA, 0x1D15E, 0x1D165, 0x1D1BB, 0x1D1C1, 0x1D400, 0x1D455, 0x1D456, + 0x1D49D, 0x1D49E, 0x1D4A0, 0x1D4A2, 0x1D4A3, 0x1D4A5, 0x1D4A7, 0x1D4A9, 0x1D4AD, 0x1D4AE, + 0x1D4BA, 0x1D4BB, 0x1D4BC, 0x1D4BD, 0x1D4C4, 0x1D4C5, 0x1D506, 0x1D507, 0x1D50B, 0x1D50D, + 0x1D515, 0x1D516, 0x1D51D, 0x1D51E, 0x1D53A, 0x1D53B, 0x1D53F, 0x1D540, 0x1D545, 0x1D546, + 0x1D547, 0x1D54A, 0x1D551, 0x1D552, 0x1D6A6, 0x1D6A8, 0x1D7CC, 0x1D7CE, 0x1D800, 0x1E030, + 0x1E051, 0x1E06B, 0x1E06E, 0x1EE00, 0x1EE04, 0x1EE05, 0x1EE20, 0x1EE21, 0x1EE23, 0x1EE24, + 0x1EE25, 0x1EE27, 0x1EE28, 0x1EE29, 0x1EE33, 0x1EE34, 0x1EE38, 0x1EE39, 0x1EE3A, 0x1EE3B, + 0x1EE3C, 0x1EE42, 0x1EE43, 0x1EE47, 0x1EE48, 0x1EE49, 0x1EE4A, 0x1EE4B, 0x1EE4C, 0x1EE4D, + 0x1EE50, 0x1EE51, 0x1EE53, 0x1EE54, 0x1EE55, 0x1EE57, 0x1EE58, 0x1EE59, 0x1EE5A, 0x1EE5B, + 0x1EE5C, 0x1EE5D, 0x1EE5E, 0x1EE5F, 0x1EE60, 0x1EE61, 0x1EE63, 0x1EE64, 0x1EE65, 0x1EE67, + 0x1EE6B, 0x1EE6C, 0x1EE73, 0x1EE74, 0x1EE78, 0x1EE79, 0x1EE7D, 0x1EE7E, 0x1EE7F, 0x1EE80, + 0x1EE8A, 0x1EE8B, 0x1EE9C, 0x1EEA1, 0x1EEA4, 0x1EEA5, 0x1EEAA, 0x1EEAB, 0x1EEBC, 0x1F100, + 0x1F10B, 0x1F110, 0x1F12B, 0x1F12F, 0x1F130, 0x1F150, 0x1F16A, 0x1F16D, 0x1F190, 0x1F191, + 0x1F200, 0x1F203, 0x1F210, 0x1F23C, 0x1F240, 0x1F249, 0x1F250, 0x1F252, 0x1FBF0, 0x1FBFA, + 0x2F800, 0x2FA1E, + }; + + private static final byte[] VALUES = { + 11, 10, 11, 1, 11, 15, 11, 1, 11, 15, 1, 11, 1, 15, 11, 5, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 1, 0, 11, 0, 1, 11, 0, 1, 11, 0, 11, 0, 11, 0, 1, + 11, 0, 11, 0, 11, 1, 0, 11, 0, 11, 0, 1, 0, 11, 0, 11, 0, 11, + 0, 11, 15, 11, 1, 11, 15, 11, 0, 11, 0, 11, 0, 11, 1, 11, 0, 11, + 1, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 1, 0, 1, 11, 1, 11, 1, + 11, 1, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 1, 11, 0, 11, 1, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 1, 11, 1, 11, 1, 11, 10, 11, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 1, 0, 1, 11, 0, 11, + 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 15, 11, 0, 11, + 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, + 15, 11, 15, 11, 15, 11, 15, 14, 11, 15, 11, 15, 11, 0, 1, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 0, 1, 0, 1, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 1, 11, 0, 1, 10, 1, 11, 10, 11, 1, 11, 1, 11, 10, 11, 1, 11, 1, + 11, 1, 11, 1, 11, 1, 11, 1, 11, 1, 11, 15, 11, 15, 14, 11, 14, 11, + 1, 11, 1, 4, 1, 11, 1, 11, 1, 4, 11, 4, 1, 11, 4, 11, 15, 1, + 15, 11, 4, 11, 0, 11, 4, 11, 0, 4, 11, 4, 11, 4, 1, 4, 11, 1, + 4, 11, 4, 11, 5, 1, 11, 5, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 1, 11, 1, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 2, 1, 2, 11, 1, 11, 1, 11, 0, 11, 14, + 15, 11, 15, 11, 1, 11, 1, 11, 1, 11, 17, 11, 1, 11, 1, 11, 0, 11, + 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, + 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, + 0, 11, 0, 11, 1, 11, 0, 16, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 16, 11, 1, 11, 15, 11, 1, 11, 1, 2, 11, 13, 2, 11, 2, 1, 13, 2, + 13, 1, 13, 1, 13, 11, 15, 11, 15, 11, 15, 11, 15, 11, 15, 11, 15, 11, + 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, + 0, 11, 1, 11, 1, 11, 0, 11, 0, 4, 0, 11, 0, 11, 0, 11, 0, 11, + 0, 11, 0, 1, 7, 3, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, + 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, + 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, + 7, 3, 7, 3, 7, 3, 7, 3, 7, 3, 7, 3, 7, 3, 6, 8, 7, 3, + 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 7, 3, 6, 8, 7, 3, + 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 7, 3, 11, 7, 3, 6, 8, 7, + 3, 7, 3, 7, 3, 7, 3, 7, 3, 7, 3, 7, 3, 6, 8, 6, 8, 7, + 3, 7, 3, 7, 3, 7, 3, 7, 3, 7, 3, 7, 3, 6, 7, 3, 6, 7, + 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, 3, 7, 11, 6, 3, 6, 3, + 6, 3, 6, 3, 6, 3, 6, 3, 6, 3, 6, 3, 6, 3, 6, 3, 6, 3, + 6, 3, 6, 3, 6, 3, 6, 3, 6, 3, 6, 3, 6, 11, 6, 3, 6, 3, + 6, 3, 6, 3, 6, 3, 6, 3, 6, 3, 11, 7, 11, 16, 11, 16, 11, 16, + 1, 12, 11, 12, 11, 12, 11, 7, 8, 7, 11, 7, 11, 7, 8, 7, 8, 7, + 8, 7, 8, 7, 8, 7, 3, 7, 3, 7, 3, 7, 3, 7, 3, 6, 8, 7, + 3, 7, 3, 6, 8, 7, 3, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, + 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 7, 3, 7, 3, 7, 3, 7, + 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, + 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, + 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, 8, 7, 3, 6, + 8, 7, 3, 6, 8, 7, 3, 7, 3, 7, 3, 6, 8, 7, 3, 7, 3, 7, + 3, 7, 3, 11, 17, 9, 11, 9, 11, 9, 11, 9, 11, 9, 11, 17, 11, 9, + 11, 0, 11, 0, 11, 15, 11, 15, 11, 15, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, 11, 0, + 11, 0, 11, 0, 11, 0, 11, 0, 11, 4, 11, 0, 11, 0, 11, 4, 11, 4, + 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, + 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, + 11, 4, 11, 15, 14, 15, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, + 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, + 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, + 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, 11, 4, + 11, 1, 11, 1, 2, 11, 13, 11, 15, 11, 13, 11, 13, 11, 13, 11, 1, 11, + 2, 11, 4, 11, 0, 11, + }; + + static byte propertyOf(int codePoint) { + if (codePoint < 0 || codePoint > 0x10ffff) { + throw new IllegalArgumentException("Not a Unicode code point: " + codePoint); + } + int low = 0; + int high = STARTS.length - 1; + while (low <= high) { + int middle = (low + high) >>> 1; + if (STARTS[middle] <= codePoint) { + low = middle + 1; + } else { + high = middle - 1; + } + } + return VALUES[high]; + } + + static int rangeCount() { + return STARTS.length; + } + + static int rangeStart(int index) { + return STARTS[index]; + } + + static int rangeEnd(int index) { + return index + 1 < STARTS.length ? STARTS[index + 1] - 1 : 0x10ffff; + } + + static byte rangeValue(int index) { + return VALUES[index]; + } + + static boolean isPropertyAlias(String alias) { + boolean hasIsPrefix = alias != null && alias.startsWith("Is"); + String normalized = loose(hasIsPrefix ? alias.substring(2) : alias); + switch (normalized) { + case "decompositiontype": + case "dt": + return true; + default: + return false; + } + } + + static byte valueForAlias(String alias) { + String normalized = loose(alias); + switch (normalized) { + case "can": + case "canonical": + return CANONICAL; + case "com": + case "compat": + return COMPAT; + case "circle": + case "enc": + return CIRCLE; + case "fin": + case "final": + return FINAL; + case "font": + return FONT; + case "fra": + case "fraction": + return FRACTION; + case "init": + case "initial": + return INITIAL; + case "iso": + case "isolated": + return ISOLATED; + case "med": + case "medial": + return MEDIAL; + case "nar": + case "narrow": + return NARROW; + case "nb": + case "nobreak": + return NOBREAK; + case "none": + return NONE; + case "small": + case "sml": + return SMALL; + case "sqr": + case "square": + return SQUARE; + case "sub": + return SUB; + case "sup": + case "super": + return SUPER; + case "vert": + case "vertical": + return VERTICAL; + case "wide": + return WIDE; + case "noncanon": + case "noncanonical": + return NON_CANONICAL; + default: + return INVALID; + } + } + + static String canonicalValueName(byte value) { + return value >= 0 && value < CANONICAL_NAMES.length + ? CANONICAL_NAMES[value] : null; + } + + static boolean matches(byte property, byte requested) { + if (requested == NON_CANONICAL) { + return property != CANONICAL && property != NONE; + } + return property == requested; + } + + private static String loose(String alias) { + if (alias == null) return ""; + StringBuilder normalized = new StringBuilder(alias.length()); + for (int i = 0; i < alias.length(); i++) { + char character = alias.charAt(i); + if (character == '_' || character == '-' || character == ' ' + || (character >= '\t' && character <= '\r')) continue; + normalized.append(Character.toLowerCase(character)); + } + return normalized.toString(); + } + + private PerlUnicodeDecompositionTypeData() { + } +} diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index f2ee514a9b..d6cf6dfbe6 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -17,6 +17,8 @@ public class UnicodeResolver { private static final Pattern USER_DEFINED_PROPERTY_NAME = Pattern.compile( "^(?:[A-Za-z_][A-Za-z0-9_]*::)*(?:Is|In)[A-Za-z_][A-Za-z0-9_]*$"); + private static final UnicodeSet[] PERL_DECOMPOSITION_TYPE_SETS = + buildPerlDecompositionTypeSets(); /** * Cache for user-defined property subroutine results. @@ -971,6 +973,18 @@ && isCanonicalCombiningClassProperty(alias.substring(0, assignment))) { } return bidiClass; } + if (assignment > 0 && assignment < alias.length() - 1 + && PerlUnicodeDecompositionTypeData.isPropertyAlias( + alias.substring(0, assignment))) { + byte value = PerlUnicodeDecompositionTypeData.valueForAlias( + alias.substring(assignment + 1)); + if (value == PerlUnicodeDecompositionTypeData.INVALID) { + throw new IllegalArgumentException( + "Unsupported Decomposition_Type value: " + + alias.substring(assignment + 1).trim()); + } + return PERL_DECOMPOSITION_TYPE_SETS[value]; + } if (assignment > 0 && assignment < alias.length() - 1) { Boolean value = perlBooleanPropertyValue(alias.substring(assignment + 1)); if (value != null) { @@ -1044,6 +1058,24 @@ private static boolean isCanonicalCombiningClassProperty(String property) { }; } + private static UnicodeSet[] buildPerlDecompositionTypeSets() { + UnicodeSet[] sets = new UnicodeSet[ + PerlUnicodeDecompositionTypeData.NON_CANONICAL + 1]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + for (int i = 0; i < PerlUnicodeDecompositionTypeData.rangeCount(); i++) { + int start = PerlUnicodeDecompositionTypeData.rangeStart(i); + int end = PerlUnicodeDecompositionTypeData.rangeEnd(i); + byte value = PerlUnicodeDecompositionTypeData.rangeValue(i); + sets[value].add(start, end); + if (PerlUnicodeDecompositionTypeData.matches( + value, PerlUnicodeDecompositionTypeData.NON_CANONICAL)) { + sets[PerlUnicodeDecompositionTypeData.NON_CANONICAL].add(start, end); + } + } + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + private static int propertyValueDelimiter(String property) { int equals = property.indexOf('='); if (equals >= 0) return equals; @@ -1091,8 +1123,14 @@ private static String normalizePerlIsPropertyAssignment(String property) { } String name = property.substring(0, delimiter); + // Perl's assignment prefix is exactly "Is". Property and value + // aliases use loose matching after that prefix, but the prefix itself + // remains case-sensitive (for example, is_dt=Can is invalid). + if (name.length() < 2 || name.charAt(0) != 'I' || name.charAt(1) != 's') { + return property; + } String looseName = loosePropertyName(name); - if (!looseName.startsWith("is") || looseName.length() == 2) { + if (looseName.length() == 2) { return property; } diff --git a/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeDecompositionTypeDataTest.java b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeDecompositionTypeDataTest.java new file mode 100644 index 0000000000..203a100f39 --- /dev/null +++ b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeDecompositionTypeDataTest.java @@ -0,0 +1,192 @@ +package org.perlonjava.runtime.regex; + +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; + +import java.io.ByteArrayOutputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; + +@Tag("unit") +class PerlUnicodeDecompositionTypeDataTest { + private static final byte[] VALUES = { + PerlUnicodeDecompositionTypeData.CANONICAL, + PerlUnicodeDecompositionTypeData.COMPAT, + PerlUnicodeDecompositionTypeData.CIRCLE, + PerlUnicodeDecompositionTypeData.FINAL, + PerlUnicodeDecompositionTypeData.FONT, + PerlUnicodeDecompositionTypeData.FRACTION, + PerlUnicodeDecompositionTypeData.INITIAL, + PerlUnicodeDecompositionTypeData.ISOLATED, + PerlUnicodeDecompositionTypeData.MEDIAL, + PerlUnicodeDecompositionTypeData.NARROW, + PerlUnicodeDecompositionTypeData.NOBREAK, + PerlUnicodeDecompositionTypeData.NONE, + PerlUnicodeDecompositionTypeData.SMALL, + PerlUnicodeDecompositionTypeData.SQUARE, + PerlUnicodeDecompositionTypeData.SUB, + PerlUnicodeDecompositionTypeData.SUPER, + PerlUnicodeDecompositionTypeData.VERTICAL, + PerlUnicodeDecompositionTypeData.WIDE, + }; + + private static final long[] COUNTS = { + 13_253, 720, 240, 240, 1_230, 20, 171, 238, 82, + 122, 5, 1_097_026, 26, 286, 64, 250, 35, 104, + }; + + private static final String[][] ALIASES = { + {"Can", "Canonical", "can"}, + {"Com", "Compat", "com"}, + {"Enc", "Circle", "enc"}, + {"Fin", "Final", "fin"}, + {"Font", "Font", "font"}, + {"Fra", "Fraction", "fra"}, + {"Init", "Initial", "init"}, + {"Iso", "Isolated", "iso"}, + {"Med", "Medial", "med"}, + {"Nar", "Narrow", "nar"}, + {"Nb", "Nobreak", "nb"}, + {"None", "None", "none"}, + {"Sml", "Small", "sml"}, + {"Sqr", "Square", "sqr"}, + {"Sub", "Sub", "sub"}, + {"Sup", "Super", "sup"}, + {"Vert", "Vertical", "vert"}, + {"Wide", "Wide", "wide"}, + }; + + @Test + public void usesPinnedUnicode17ValuesAndBoundaries() { + assertEquals("17.0.0", PerlUnicodeDecompositionTypeData.UNICODE_VERSION); + assertEquals(PerlUnicodeDecompositionTypeData.NONE, PerlUnicodeDecompositionTypeData.propertyOf(0x0041)); + assertEquals(PerlUnicodeDecompositionTypeData.COMPAT, PerlUnicodeDecompositionTypeData.propertyOf(0x00A8)); + assertEquals(PerlUnicodeDecompositionTypeData.CANONICAL, PerlUnicodeDecompositionTypeData.propertyOf(0x00C0)); + assertEquals(PerlUnicodeDecompositionTypeData.CANONICAL, PerlUnicodeDecompositionTypeData.propertyOf(0xAC00)); + assertEquals(PerlUnicodeDecompositionTypeData.CANONICAL, PerlUnicodeDecompositionTypeData.propertyOf(0xD7A3)); + assertEquals(PerlUnicodeDecompositionTypeData.NONE, PerlUnicodeDecompositionTypeData.propertyOf(0xD7A4)); + assertEquals(PerlUnicodeDecompositionTypeData.NONE, PerlUnicodeDecompositionTypeData.propertyOf(0xA7F0)); + assertEquals(PerlUnicodeDecompositionTypeData.SUPER, PerlUnicodeDecompositionTypeData.propertyOf(0xA7F1)); + assertEquals(PerlUnicodeDecompositionTypeData.SUPER, PerlUnicodeDecompositionTypeData.propertyOf(0xA7F4)); + assertEquals(PerlUnicodeDecompositionTypeData.NONE, PerlUnicodeDecompositionTypeData.propertyOf(0xA7F5)); + assertEquals(PerlUnicodeDecompositionTypeData.NONE, PerlUnicodeDecompositionTypeData.propertyOf(0x10FFFF)); + assertThrows(IllegalArgumentException.class, () -> PerlUnicodeDecompositionTypeData.propertyOf(-1)); + assertThrows(IllegalArgumentException.class, () -> PerlUnicodeDecompositionTypeData.propertyOf(0x110000)); + } + + @Test + public void completelyPartitionsEveryUnicodeCodePoint() { + assertEquals(1_212, PerlUnicodeDecompositionTypeData.rangeCount()); + assertEquals(0, PerlUnicodeDecompositionTypeData.rangeStart(0)); + assertEquals(0x10FFFF, PerlUnicodeDecompositionTypeData.rangeEnd( + PerlUnicodeDecompositionTypeData.rangeCount() - 1)); + for (int index = 0; index < PerlUnicodeDecompositionTypeData.rangeCount(); index++) { + assertEquals(PerlUnicodeDecompositionTypeData.rangeValue(index), + PerlUnicodeDecompositionTypeData.propertyOf( + PerlUnicodeDecompositionTypeData.rangeStart(index))); + assertEquals(PerlUnicodeDecompositionTypeData.rangeValue(index), + PerlUnicodeDecompositionTypeData.propertyOf( + PerlUnicodeDecompositionTypeData.rangeEnd(index))); + if (index > 0) { + assertEquals(PerlUnicodeDecompositionTypeData.rangeEnd(index - 1) + 1, + PerlUnicodeDecompositionTypeData.rangeStart(index)); + assertTrue(PerlUnicodeDecompositionTypeData.rangeValue(index - 1) + != PerlUnicodeDecompositionTypeData.rangeValue(index)); + } + } + + long[] actualCounts = new long[VALUES.length]; + int transitions = 0; + byte previous = PerlUnicodeDecompositionTypeData.INVALID; + long nonCanonical = 0; + for (int codePoint = 0; codePoint <= 0x10FFFF; codePoint++) { + byte value = PerlUnicodeDecompositionTypeData.propertyOf(codePoint); + if (value < 0 || value >= VALUES.length) { + fail("invalid value at U+" + Integer.toHexString(codePoint)); + } + actualCounts[value]++; + if (value != previous) { + transitions++; + previous = value; + } + if (PerlUnicodeDecompositionTypeData.matches(value, PerlUnicodeDecompositionTypeData.NON_CANONICAL)) { + nonCanonical++; + } + } + assertArrayEquals(COUNTS, actualCounts); + assertEquals(1_212, transitions); + assertEquals(3_833, nonCanonical); + } + + @Test + public void acceptsAllLooseAliasesIncludingPerlUnionValues() { + assertTrue(PerlUnicodeDecompositionTypeData.isPropertyAlias("dt")); + assertTrue(PerlUnicodeDecompositionTypeData.isPropertyAlias("Decomposition Type")); + assertTrue(PerlUnicodeDecompositionTypeData.isPropertyAlias("Is-d_t")); + assertTrue(PerlUnicodeDecompositionTypeData.isPropertyAlias("Is Decomposition_Type")); + assertTrue(PerlUnicodeDecompositionTypeData.isPropertyAlias("Decomposition\tType")); + assertFalse(PerlUnicodeDecompositionTypeData.isPropertyAlias("is-d_t")); + assertFalse(PerlUnicodeDecompositionTypeData.isPropertyAlias("IS_DT")); + assertFalse(PerlUnicodeDecompositionTypeData.isPropertyAlias("Decomposition\u1680Type")); + assertFalse(PerlUnicodeDecompositionTypeData.isPropertyAlias("decomposition mapping")); + assertFalse(PerlUnicodeDecompositionTypeData.isPropertyAlias(null)); + + for (int id = 0; id < ALIASES.length; id++) { + for (String alias : ALIASES[id]) { + assertEquals(VALUES[id], + PerlUnicodeDecompositionTypeData.valueForAlias(alias), alias); + } + assertEquals(ALIASES[id][1], PerlUnicodeDecompositionTypeData.canonicalValueName(VALUES[id])); + } + assertEquals(PerlUnicodeDecompositionTypeData.CANONICAL, + PerlUnicodeDecompositionTypeData.valueForAlias("c_a-n")); + assertEquals(PerlUnicodeDecompositionTypeData.INVALID, + PerlUnicodeDecompositionTypeData.valueForAlias("c\u1680an")); + assertEquals(PerlUnicodeDecompositionTypeData.NON_CANONICAL, + PerlUnicodeDecompositionTypeData.valueForAlias("Non Canon")); + assertEquals(PerlUnicodeDecompositionTypeData.NON_CANONICAL, + PerlUnicodeDecompositionTypeData.valueForAlias("non-canonical")); + assertEquals("Non_Canonical", + PerlUnicodeDecompositionTypeData.canonicalValueName(PerlUnicodeDecompositionTypeData.NON_CANONICAL)); + assertEquals(PerlUnicodeDecompositionTypeData.INVALID, + PerlUnicodeDecompositionTypeData.valueForAlias("reserved")); + assertEquals(PerlUnicodeDecompositionTypeData.INVALID, + PerlUnicodeDecompositionTypeData.valueForAlias(null)); + assertTrue(PerlUnicodeDecompositionTypeData.matches( + PerlUnicodeDecompositionTypeData.COMPAT, PerlUnicodeDecompositionTypeData.NON_CANONICAL)); + assertFalse(PerlUnicodeDecompositionTypeData.matches( + PerlUnicodeDecompositionTypeData.CANONICAL, PerlUnicodeDecompositionTypeData.NON_CANONICAL)); + assertFalse(PerlUnicodeDecompositionTypeData.matches( + PerlUnicodeDecompositionTypeData.NONE, PerlUnicodeDecompositionTypeData.NON_CANONICAL)); + } + + @Test + public void generatorReproducesTheCheckedInClassByteForByte() throws Exception { + Path root = Path.of(System.getProperty("user.dir")); + Path generator = root.resolve("dev/tools/generate_perl_unicode_decomposition_type_data.pl"); + Path generated = root.resolve("src/main/java/org/perlonjava/runtime/regex/PerlUnicodeDecompositionTypeData.java"); + + Process process = new ProcessBuilder("perl", generator.toString()) + .directory(root.toFile()) + .redirectErrorStream(true) + .start(); + ByteArrayOutputStream output = new ByteArrayOutputStream(); + process.getInputStream().transferTo(output); + assertEquals(0, process.waitFor(), new String(output.toByteArray())); + assertArrayEquals(canonicalLf(Files.readAllBytes(generated)), + canonicalLf(output.toByteArray())); + } + + private static byte[] canonicalLf(byte[] bytes) { + return new String(bytes, StandardCharsets.UTF_8) + .replace("\r\n", "\n") + .getBytes(StandardCharsets.UTF_8); + } +} diff --git a/src/test/resources/unit/regex/unicode_decomposition_type_data.t b/src/test/resources/unit/regex/unicode_decomposition_type_data.t new file mode 100644 index 0000000000..910c9abd26 --- /dev/null +++ b/src/test/resources/unit/regex/unicode_decomposition_type_data.t @@ -0,0 +1,62 @@ +use strict; +use warnings; +use utf8; +use Test::More; + +my @cases = ( + [Can => Canonical => 0x00C0], + [Com => Compat => 0x00A8], + [Enc => Circle => 0x2460], + [Fin => Final => 0xFB51], + [Font => Font => 0x2102], + [Fra => Fraction => 0x00BC], + [Init => Initial => 0xFB54], + [Iso => Isolated => 0xFB50], + [Med => Medial => 0xFB55], + [Nar => Narrow => 0xFF61], + [Nb => Nobreak => 0x00A0], + [None => None => 0x0041], + [Sml => Small => 0xFE50], + [Sqr => Square => 0x3250], + [Sub => Sub => 0x1D62], + [Sup => Super => 0x00AA], + [Vert => Vertical => 0x309F], + [Wide => Wide => 0x3000], +); + +sub matches_property { + my ($property, $value, $code_point) = @_; + my $regex = eval "qr/\\p{$property=$value}/"; + return defined($regex) && chr($code_point) =~ $regex; +} + + for my $case (@cases) { + my ($short, $long, $code_point) = @$case; + ok(matches_property('dt', $short, $code_point), + "short Decomposition_Type value alias $short"); + ok(matches_property('Decomposition_Type', $long, $code_point), + "long Decomposition_Type value alias $long"); + } + + ok(matches_property('de composition-type', 'c_a-n', 0x00C0), + 'property and value aliases use Perl loose matching'); + ok(!matches_property('dt', 'None', 0x00C0), + 'the ordered missing-value default does not override explicit ranges'); + + my $none = eval 'qr/\p{dt=None}/'; + ok(defined($none) && chr(0x10FFFF) =~ $none, + 'the missing-value default covers the Unicode maximum'); + + ok(matches_property('dt', 'NonCanon', 0x00A8), + 'Perl short non-canonical union value is valid'); + ok(matches_property('Decomposition Type', 'Non_Canonical', 0x2460), + 'Perl long non-canonical union value uses loose matching'); + ok(matches_property('Is_Dt', 'Can', 0x00C0), + 'Perl Is_ prefix accepts the short property alias'); + ok(matches_property('Is Decomposition Type', 'Wide', 0x3000), + 'Perl Is_ prefix accepts the long property alias loosely'); + ok(!matches_property('is_dt', 'Can', 0x00C0), + 'Perl Is prefix remains case-sensitive'); + ok(!matches_property("Decomposition\x{1680}Type", 'Can', 0x00C0), + 'Perl loose matching does not ignore Unicode whitespace'); +done_testing; From a07effb5ade0f8dc5dd566fd61da960ad35c7b3d Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:50 +0200 Subject: [PATCH 31/60] feat(regex): add pinned Unicode East Asian widths (#1036) Squashed integration of PR #1036 at c587db9c0. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 33 ++- ...rate_perl_unicode_east_asian_width_data.pl | 266 ++++++++++++++++++ docs/reference/feature-matrix.md | 4 +- .../regex/PerlUnicodeEastAsianWidthData.java | 262 +++++++++++++++++ .../runtime/regex/UnicodeResolver.java | 20 +- .../PerlUnicodeEastAsianWidthDataTest.java | 99 +++++++ .../regex/unicode_east_asian_width_pinned.t | 50 ++++ 7 files changed, 721 insertions(+), 13 deletions(-) create mode 100644 dev/tools/generate_perl_unicode_east_asian_width_data.pl create mode 100644 src/main/java/org/perlonjava/runtime/regex/PerlUnicodeEastAsianWidthData.java create mode 100644 src/test/java/org/perlonjava/runtime/regex/PerlUnicodeEastAsianWidthDataTest.java create mode 100644 src/test/resources/unit/regex/unicode_east_asian_width_pinned.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 0e1d2c96a3..97a43b0405 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,7 +224,7 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1034, stacked +The published integration stack is preserved through draft PR #1035, stacked on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the @@ -237,9 +237,9 @@ binary `ASCII_Hex_Digit` values, pinned General_Category sets, and exact native line boundaries. PR #1031 adds pinned Canonical_Combining_Class sets and valid empty-property rendering. PR #1032 integrates native numeric escapes through U+10FFFF; PR #1033 adds pinned Bidi_Class sets; PR #1034 integrates native -vertical-whitespace escapes. The current WIP integrates Decomposition_Type; -East_Asian_Width is ready for the next focused integration, while Numeric_Value -and Joining_Group data slices advance independently. +vertical-whitespace escapes; PR #1035 adds Decomposition_Type. The current WIP +integrates East_Asian_Width; Numeric_Value and Joining_Group are ready, while +Block and Script/Script_Extensions data slices advance independently. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -338,6 +338,16 @@ gains 640 assertions to 34,892/41,843 identically on both execution backends with zero numbered regressions, raising current generated evidence to 342,656/407,367. +`East_Asian_Width`/`ea` assignments now resolve all six values from a complete +pinned Unicode 17 partition, including the ordered CJK `Wide` and general +`Neutral` missing defaults. Short/long and loose aliases are covered, and +surrogate range endpoints render as explicit Joni hex escapes rather than +lossy literal surrogates. The focused oracle passes 31/31 on system Perl, JVM, +and interpreter; protected boundary smoke remains 169/169 per backend. Chunk +01 gains 216 assertions to 35,108/41,843 identically on both execution backends +with zero numbered regressions, raising current generated evidence to +342,872/407,367. + Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while ordinary and substitution callbacks execute, without misclassifying escaped or @@ -593,6 +603,11 @@ is retained for now. value and exact `Is` prefix policy. The focused oracle passes 45/45 and chunk 01 gains 640 assertions to 34,892/41,843 on both backends with zero numbered regressions. + - [x] Generated and integrated a complete pinned Unicode 17.0 + East_Asian_Width partition with all ordered missing defaults and lossless + surrogate-range rendering. The focused oracle passes 31/31 and chunk 01 + gains 216 assertions to 35,108/41,843 on both backends with zero numbered + regressions. - [x] Integrated native Perl `\v`/`\V` dispatch inside and outside character classes (`1eff1db97`, integrated as `6328935cd`). The focused oracle passes 92/92 and unchanged `reg_posixcc.t` passes 2,560/2,560 on both backends. @@ -638,14 +653,14 @@ is retained for now. ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1034. Publish the validated - Decomposition_Type integration in a separate focused WIP PR, then integrate - East_Asian_Width without squashing the independently reviewed data commit. +1. Land review-ready PR #1024 and draft PRs #1025–#1035. Publish the validated + East_Asian_Width integration in a separate focused WIP PR, preserving the + independently reviewed data commit. 2. Preserve the now-complete native Joni boundary corpus at 239,866/239,866: sentence chunk 05, line chunks 06–09, and word chunk 10 must remain exact on JVM and interpreter while property and parser work continues. -3. Integrate the independently generated East_Asian_Width, Numeric_Value, and - Joining_Group slices, then continue the measured residual order: binary +3. Integrate the independently generated Numeric_Value and Joining_Group + slices, then continue the measured residual order: binary property values, Block, Script/Script_Extensions, and break-property values. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. diff --git a/dev/tools/generate_perl_unicode_east_asian_width_data.pl b/dev/tools/generate_perl_unicode_east_asian_width_data.pl new file mode 100644 index 0000000000..d2fd0137d7 --- /dev/null +++ b/dev/tools/generate_perl_unicode_east_asian_width_data.pl @@ -0,0 +1,266 @@ +#!/usr/bin/env perl +use strict; +use warnings; +use Digest::SHA qw(sha256_hex); +use File::Spec; +use FindBin; + +my $expected_version = '17.0.0'; +my $root = File::Spec->catdir($FindBin::Bin, '..', '..'); +my $unicore = File::Spec->catdir($root, 'perl5', 'lib', 'unicore'); +my @sources = ( + { + name => 'DerivedEastAsianWidth-17.0.0.txt', + path => File::Spec->catfile($unicore, 'extracted', 'DEastAsianWidth.txt'), + hash => '0b5523a2217cb318d20b329a05d31eec5af5686ba09d263b85bb75a28989a3a8', + version => qr/^# DerivedEastAsianWidth-\Q$expected_version\E\.txt$/m, + }, + { + name => 'PropertyValueAliases-17.0.0.txt', + path => File::Spec->catfile($unicore, 'PropValueAliases.txt'), + hash => '670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01', + version => qr/^# PropertyValueAliases-\Q$expected_version\E\.txt$/m, + }, + { + name => 'PropertyAliases-17.0.0.txt', + path => File::Spec->catfile($unicore, 'PropertyAliases.txt'), + hash => '4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb', + version => qr/^# PropertyAliases-\Q$expected_version\E\.txt$/m, + }, +); + +sub read_source { + my ($source) = @_; + open my $input, '<:raw', $source->{path} + or die "Cannot read $source->{path}: $!\n"; + local $/; + my $text = <$input>; + close $input or die "Cannot close $source->{path}: $!\n"; + my $actual_hash = sha256_hex($text); + die "$source->{path} SHA-256 mismatch: expected $source->{hash}, found $actual_hash\n" + unless $actual_hash eq $source->{hash}; + die "$source->{path} is not pinned Unicode $expected_version data\n" + unless $text =~ $source->{version}; + $source->{text} = $text; +} + +sub trim { + my ($text) = @_; + $text =~ s/^\s+|\s+$//g; + return $text; +} + +sub loose_name { + my ($name) = @_; + $name = lc $name; + $name =~ s/[\s_-]+//g; + return $name; +} + +sub parse_range { + my ($text) = @_; + my ($start, $end) = split /\.\./, $text; + return (hex($start), hex(defined $end ? $end : $start)); +} + +read_source($_) for @sources; + +my $version_path = File::Spec->catfile($unicore, 'version'); +open my $version_input, '<', $version_path or die "Cannot read $version_path: $!\n"; +chomp(my $unicode_version = <$version_input>); +close $version_input or die "Cannot close $version_path: $!\n"; +die "Expected Unicode $expected_version, found $unicode_version\n" + unless $unicode_version eq $expected_version; + +my (@short_values, @long_values, %alias_index); +for my $line (split /\n/, $sources[1]{text}) { + next if $line =~ /^\s*#/; + $line =~ s/#.*$//; + my @fields = map { trim($_) } split /;/, $line; + next unless @fields >= 3 && $fields[0] eq 'ea'; + my $index = scalar @short_values; + push @short_values, $fields[1]; + push @long_values, $fields[2]; + for my $alias (@fields[1 .. $#fields]) { + next unless length $alias; + my $loose = loose_name($alias); + die "East_Asian_Width alias collision for '$alias'\n" + if exists $alias_index{$loose} && $alias_index{$loose} != $index; + $alias_index{$loose} = $index; + } +} +die "Expected 6 East_Asian_Width values, found " . scalar(@short_values) . "\n" + unless @short_values == 6; + +my %property_aliases; +for my $line (split /\n/, $sources[2]{text}) { + next if $line =~ /^\s*#/; + $line =~ s/#.*$//; + my @fields = map { trim($_) } split /;/, $line; + next unless @fields >= 2 && ($fields[0] eq 'ea' || $fields[1] eq 'East_Asian_Width'); + $property_aliases{loose_name($_)} = 1 for grep { length } @fields; +} +die "Pinned property aliases do not define ea and East_Asian_Width\n" + unless $property_aliases{ea} && $property_aliases{eastasianwidth}; + +my (@missing, @explicit); +for my $line (split /\n/, $sources[0]{text}) { + if ($line =~ /^#\s*\@missing:\s*([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z_]+)/) { + my ($range, $value) = ($1, $2); + my ($start, $end) = parse_range($range); + my $index = $alias_index{loose_name($value)}; + die "Unknown \@missing East_Asian_Width value '$value'\n" unless defined $index; + push @missing, [$start, $end, $index]; + next; + } + next unless $line =~ /^([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z_]+)/; + my ($range, $value) = ($1, $2); + my ($start, $end) = parse_range($range); + my $index = $alias_index{loose_name($value)}; + die "Unknown explicit East_Asian_Width value '$value'\n" unless defined $index; + push @explicit, [$start, $end, $index]; +} +die "East_Asian_Width data has no ranges or defaults\n" unless @explicit && @missing; + +my $default_index = $alias_index{loose_name('Neutral')}; +my @code_value = ($default_index) x 0x110000; +for my $range (@missing) { + my ($code, $end, $index) = @$range; + $code_value[$code++] = $index while $code <= $end; +} +for my $range (@explicit) { + my ($code, $end, $index) = @$range; + $code_value[$code++] = $index while $code <= $end; +} + +my @ranges; +my ($range_start, $range_value) = (0, $code_value[0]); +for my $code (1 .. 0x10ffff) { + next if $code_value[$code] == $range_value; + push @ranges, [$range_start, $code - 1, $range_value]; + ($range_start, $range_value) = ($code, $code_value[$code]); +} +push @ranges, [$range_start, 0x10ffff, $range_value]; + +print <<'HEADER'; +/* + * Generated from Perl 5.44's pinned Unicode Character Database. Do not edit manually. + * +HEADER +for my $source (@sources) { + print " * Source: $source->{name}\n"; + for my $line (split /\n/, $source->{text}) { + next unless $line =~ /^# (?:©|Unicode and|the U\.S\.|For terms of use and license)/; + $line =~ s/^# / * /; + print "$line\n"; + } + print " *\n"; +} +print <<'HEADER_END'; + */ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; + +final class PerlUnicodeEastAsianWidthData { +HEADER_END + +print " static final String UNICODE_VERSION = \"$unicode_version\";\n"; +print " static final String DEAST_ASIAN_WIDTH_SHA256 = \"$sources[0]{hash}\";\n"; +print " static final String PROP_VALUE_ALIASES_SHA256 = \"$sources[1]{hash}\";\n"; +print " static final String PROPERTY_ALIASES_SHA256 = \"$sources[2]{hash}\";\n\n"; + +print " private static final String[] SHORT_VALUES = {\n "; +print join(', ', map { qq{"$_"} } @short_values); +print "\n };\n"; +print " private static final String[] LONG_VALUES = {\n "; +print join(', ', map { qq{"$_"} } @long_values); +print "\n };\n"; + +my @aliases = sort keys %alias_index; +print " private static final String[] VALUE_ALIASES = {\n "; +print join(', ', map { qq{"$_"} } @aliases); +print "\n };\n"; +print " private static final byte[] VALUE_ALIAS_INDEX = {\n "; +print join(', ', map { $alias_index{$_} } @aliases); +print "\n };\n"; + +my @property_aliases = sort keys %property_aliases; +print " private static final String[] PROPERTY_ALIASES = {\n "; +print join(', ', map { qq{"$_"} } @property_aliases); +print "\n };\n\n"; + +print " private static final int[] RANGES = {\n"; +for (my $i = 0; $i < @ranges; $i += 4) { + my $end = $i + 3 < $#ranges ? $i + 3 : $#ranges; + print " ", join(', ', map { + sprintf '0x%X, 0x%X, %d', @{$ranges[$_]}[0, 1, 2] + } $i .. $end), ",\n"; +} +print <<'FOOTER'; + }; + + private static final UnicodeSet[] VALUE_SETS = buildValueSets(); + + static boolean isPropertyAlias(String alias) { + String loose = looseName(alias); + if (loose == null) return false; + for (String candidate : PROPERTY_ALIASES) { + if (candidate.equals(loose)) return true; + } + return false; + } + + static UnicodeSet valueSet(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : VALUE_SETS[index]; + } + + static String shortValue(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : SHORT_VALUES[index]; + } + + static String canonicalValue(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : LONG_VALUES[index]; + } + + static String[] canonicalValues() { + return LONG_VALUES.clone(); + } + + private static int valueIndex(String alias) { + String loose = looseName(alias); + if (loose == null) return -1; + for (int i = 0; i < VALUE_ALIASES.length; i++) { + if (VALUE_ALIASES[i].equals(loose)) return VALUE_ALIAS_INDEX[i]; + } + return -1; + } + + private static String looseName(String name) { + if (name == null) return null; + StringBuilder loose = new StringBuilder(name.length()); + for (int i = 0; i < name.length(); i++) { + char character = name.charAt(i); + if (character == '_' || character == '-' || Character.isWhitespace(character)) continue; + loose.append(Character.toLowerCase(character)); + } + return loose.toString(); + } + + private static UnicodeSet[] buildValueSets() { + UnicodeSet[] sets = new UnicodeSet[LONG_VALUES.length]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + for (int i = 0; i < RANGES.length; i += 3) { + sets[RANGES[i + 2]].add(RANGES[i], RANGES[i + 1]); + } + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + + private PerlUnicodeEastAsianWidthData() { + } +} +FOOTER diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index f9a8158547..94c2389a2b 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -379,7 +379,7 @@ my @copy = @{$z}; # ERROR - ✅ **Backreferences to Named Groups**: Using `\k` or `\g{name}` for backreferences to named groups is supported. - ✅ **Relative Backreferences**: Using `\g{-n}` for relative backreferences. - ✅ **Basic Unicode Properties**: Common `\p{...}` and `\P{...}` forms such as `\p{L}` execute through Joni. -- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, and Decomposition_Type assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, and reserved or composite values; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. +- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, and East_Asian_Width assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, and reserved or composite values; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. - ✅ **Possessive Quantifiers**: Quantifiers like `*+`, `++`, `?+`, and `{n,m}+`, which disable backtracking, are supported. - ✅ **Atomic Grouping**: Use of `(?>...)` for atomic groups is supported. - ✅ **`\K` assertion**: Keep left — in `s///`, text before `\K` is preserved; match variables reflect only the portion after `\K`. @@ -398,7 +398,7 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 342,656/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 342,872/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. - ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. - ✅ **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter. - ✅ **Unicode Word Boundaries**: Native `\b{wb}`/`\B{wb}` implement WB1–WB16 and WB999 from reproducible Perl 5.44 Unicode 17.0 word-break and extended-pictographic tables. Authoritative chunk 10 passes 19,510/19,510 identically on JVM and interpreter. diff --git a/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeEastAsianWidthData.java b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeEastAsianWidthData.java new file mode 100644 index 0000000000..72c8236559 --- /dev/null +++ b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeEastAsianWidthData.java @@ -0,0 +1,262 @@ +/* + * Generated from Perl 5.44's pinned Unicode Character Database. Do not edit manually. + * + * Source: DerivedEastAsianWidth-17.0.0.txt + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + * + * Source: PropertyValueAliases-17.0.0.txt + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + * + * Source: PropertyAliases-17.0.0.txt + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + * + */ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; + +final class PerlUnicodeEastAsianWidthData { + static final String UNICODE_VERSION = "17.0.0"; + static final String DEAST_ASIAN_WIDTH_SHA256 = "0b5523a2217cb318d20b329a05d31eec5af5686ba09d263b85bb75a28989a3a8"; + static final String PROP_VALUE_ALIASES_SHA256 = "670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01"; + static final String PROPERTY_ALIASES_SHA256 = "4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb"; + + private static final String[] SHORT_VALUES = { + "A", "F", "H", "N", "Na", "W" + }; + private static final String[] LONG_VALUES = { + "Ambiguous", "Fullwidth", "Halfwidth", "Neutral", "Narrow", "Wide" + }; + private static final String[] VALUE_ALIASES = { + "a", "ambiguous", "f", "fullwidth", "h", "halfwidth", "n", "na", "narrow", "neutral", "w", "wide" + }; + private static final byte[] VALUE_ALIAS_INDEX = { + 0, 0, 1, 1, 2, 2, 3, 4, 4, 3, 5, 5 + }; + private static final String[] PROPERTY_ALIASES = { + "ea", "eastasianwidth" + }; + + private static final int[] RANGES = { + 0x0, 0x1F, 3, 0x20, 0x7E, 4, 0x7F, 0xA0, 3, 0xA1, 0xA1, 0, + 0xA2, 0xA3, 4, 0xA4, 0xA4, 0, 0xA5, 0xA6, 4, 0xA7, 0xA8, 0, + 0xA9, 0xA9, 3, 0xAA, 0xAA, 0, 0xAB, 0xAB, 3, 0xAC, 0xAC, 4, + 0xAD, 0xAE, 0, 0xAF, 0xAF, 4, 0xB0, 0xB4, 0, 0xB5, 0xB5, 3, + 0xB6, 0xBA, 0, 0xBB, 0xBB, 3, 0xBC, 0xBF, 0, 0xC0, 0xC5, 3, + 0xC6, 0xC6, 0, 0xC7, 0xCF, 3, 0xD0, 0xD0, 0, 0xD1, 0xD6, 3, + 0xD7, 0xD8, 0, 0xD9, 0xDD, 3, 0xDE, 0xE1, 0, 0xE2, 0xE5, 3, + 0xE6, 0xE6, 0, 0xE7, 0xE7, 3, 0xE8, 0xEA, 0, 0xEB, 0xEB, 3, + 0xEC, 0xED, 0, 0xEE, 0xEF, 3, 0xF0, 0xF0, 0, 0xF1, 0xF1, 3, + 0xF2, 0xF3, 0, 0xF4, 0xF6, 3, 0xF7, 0xFA, 0, 0xFB, 0xFB, 3, + 0xFC, 0xFC, 0, 0xFD, 0xFD, 3, 0xFE, 0xFE, 0, 0xFF, 0x100, 3, + 0x101, 0x101, 0, 0x102, 0x110, 3, 0x111, 0x111, 0, 0x112, 0x112, 3, + 0x113, 0x113, 0, 0x114, 0x11A, 3, 0x11B, 0x11B, 0, 0x11C, 0x125, 3, + 0x126, 0x127, 0, 0x128, 0x12A, 3, 0x12B, 0x12B, 0, 0x12C, 0x130, 3, + 0x131, 0x133, 0, 0x134, 0x137, 3, 0x138, 0x138, 0, 0x139, 0x13E, 3, + 0x13F, 0x142, 0, 0x143, 0x143, 3, 0x144, 0x144, 0, 0x145, 0x147, 3, + 0x148, 0x14B, 0, 0x14C, 0x14C, 3, 0x14D, 0x14D, 0, 0x14E, 0x151, 3, + 0x152, 0x153, 0, 0x154, 0x165, 3, 0x166, 0x167, 0, 0x168, 0x16A, 3, + 0x16B, 0x16B, 0, 0x16C, 0x1CD, 3, 0x1CE, 0x1CE, 0, 0x1CF, 0x1CF, 3, + 0x1D0, 0x1D0, 0, 0x1D1, 0x1D1, 3, 0x1D2, 0x1D2, 0, 0x1D3, 0x1D3, 3, + 0x1D4, 0x1D4, 0, 0x1D5, 0x1D5, 3, 0x1D6, 0x1D6, 0, 0x1D7, 0x1D7, 3, + 0x1D8, 0x1D8, 0, 0x1D9, 0x1D9, 3, 0x1DA, 0x1DA, 0, 0x1DB, 0x1DB, 3, + 0x1DC, 0x1DC, 0, 0x1DD, 0x250, 3, 0x251, 0x251, 0, 0x252, 0x260, 3, + 0x261, 0x261, 0, 0x262, 0x2C3, 3, 0x2C4, 0x2C4, 0, 0x2C5, 0x2C6, 3, + 0x2C7, 0x2C7, 0, 0x2C8, 0x2C8, 3, 0x2C9, 0x2CB, 0, 0x2CC, 0x2CC, 3, + 0x2CD, 0x2CD, 0, 0x2CE, 0x2CF, 3, 0x2D0, 0x2D0, 0, 0x2D1, 0x2D7, 3, + 0x2D8, 0x2DB, 0, 0x2DC, 0x2DC, 3, 0x2DD, 0x2DD, 0, 0x2DE, 0x2DE, 3, + 0x2DF, 0x2DF, 0, 0x2E0, 0x2FF, 3, 0x300, 0x36F, 0, 0x370, 0x390, 3, + 0x391, 0x3A1, 0, 0x3A2, 0x3A2, 3, 0x3A3, 0x3A9, 0, 0x3AA, 0x3B0, 3, + 0x3B1, 0x3C1, 0, 0x3C2, 0x3C2, 3, 0x3C3, 0x3C9, 0, 0x3CA, 0x400, 3, + 0x401, 0x401, 0, 0x402, 0x40F, 3, 0x410, 0x44F, 0, 0x450, 0x450, 3, + 0x451, 0x451, 0, 0x452, 0x10FF, 3, 0x1100, 0x115F, 5, 0x1160, 0x200F, 3, + 0x2010, 0x2010, 0, 0x2011, 0x2012, 3, 0x2013, 0x2016, 0, 0x2017, 0x2017, 3, + 0x2018, 0x2019, 0, 0x201A, 0x201B, 3, 0x201C, 0x201D, 0, 0x201E, 0x201F, 3, + 0x2020, 0x2022, 0, 0x2023, 0x2023, 3, 0x2024, 0x2027, 0, 0x2028, 0x202F, 3, + 0x2030, 0x2030, 0, 0x2031, 0x2031, 3, 0x2032, 0x2033, 0, 0x2034, 0x2034, 3, + 0x2035, 0x2035, 0, 0x2036, 0x203A, 3, 0x203B, 0x203B, 0, 0x203C, 0x203D, 3, + 0x203E, 0x203E, 0, 0x203F, 0x2073, 3, 0x2074, 0x2074, 0, 0x2075, 0x207E, 3, + 0x207F, 0x207F, 0, 0x2080, 0x2080, 3, 0x2081, 0x2084, 0, 0x2085, 0x20A8, 3, + 0x20A9, 0x20A9, 2, 0x20AA, 0x20AB, 3, 0x20AC, 0x20AC, 0, 0x20AD, 0x2102, 3, + 0x2103, 0x2103, 0, 0x2104, 0x2104, 3, 0x2105, 0x2105, 0, 0x2106, 0x2108, 3, + 0x2109, 0x2109, 0, 0x210A, 0x2112, 3, 0x2113, 0x2113, 0, 0x2114, 0x2115, 3, + 0x2116, 0x2116, 0, 0x2117, 0x2120, 3, 0x2121, 0x2122, 0, 0x2123, 0x2125, 3, + 0x2126, 0x2126, 0, 0x2127, 0x212A, 3, 0x212B, 0x212B, 0, 0x212C, 0x2152, 3, + 0x2153, 0x2154, 0, 0x2155, 0x215A, 3, 0x215B, 0x215E, 0, 0x215F, 0x215F, 3, + 0x2160, 0x216B, 0, 0x216C, 0x216F, 3, 0x2170, 0x2179, 0, 0x217A, 0x2188, 3, + 0x2189, 0x2189, 0, 0x218A, 0x218F, 3, 0x2190, 0x2199, 0, 0x219A, 0x21B7, 3, + 0x21B8, 0x21B9, 0, 0x21BA, 0x21D1, 3, 0x21D2, 0x21D2, 0, 0x21D3, 0x21D3, 3, + 0x21D4, 0x21D4, 0, 0x21D5, 0x21E6, 3, 0x21E7, 0x21E7, 0, 0x21E8, 0x21FF, 3, + 0x2200, 0x2200, 0, 0x2201, 0x2201, 3, 0x2202, 0x2203, 0, 0x2204, 0x2206, 3, + 0x2207, 0x2208, 0, 0x2209, 0x220A, 3, 0x220B, 0x220B, 0, 0x220C, 0x220E, 3, + 0x220F, 0x220F, 0, 0x2210, 0x2210, 3, 0x2211, 0x2211, 0, 0x2212, 0x2214, 3, + 0x2215, 0x2215, 0, 0x2216, 0x2219, 3, 0x221A, 0x221A, 0, 0x221B, 0x221C, 3, + 0x221D, 0x2220, 0, 0x2221, 0x2222, 3, 0x2223, 0x2223, 0, 0x2224, 0x2224, 3, + 0x2225, 0x2225, 0, 0x2226, 0x2226, 3, 0x2227, 0x222C, 0, 0x222D, 0x222D, 3, + 0x222E, 0x222E, 0, 0x222F, 0x2233, 3, 0x2234, 0x2237, 0, 0x2238, 0x223B, 3, + 0x223C, 0x223D, 0, 0x223E, 0x2247, 3, 0x2248, 0x2248, 0, 0x2249, 0x224B, 3, + 0x224C, 0x224C, 0, 0x224D, 0x2251, 3, 0x2252, 0x2252, 0, 0x2253, 0x225F, 3, + 0x2260, 0x2261, 0, 0x2262, 0x2263, 3, 0x2264, 0x2267, 0, 0x2268, 0x2269, 3, + 0x226A, 0x226B, 0, 0x226C, 0x226D, 3, 0x226E, 0x226F, 0, 0x2270, 0x2281, 3, + 0x2282, 0x2283, 0, 0x2284, 0x2285, 3, 0x2286, 0x2287, 0, 0x2288, 0x2294, 3, + 0x2295, 0x2295, 0, 0x2296, 0x2298, 3, 0x2299, 0x2299, 0, 0x229A, 0x22A4, 3, + 0x22A5, 0x22A5, 0, 0x22A6, 0x22BE, 3, 0x22BF, 0x22BF, 0, 0x22C0, 0x2311, 3, + 0x2312, 0x2312, 0, 0x2313, 0x2319, 3, 0x231A, 0x231B, 5, 0x231C, 0x2328, 3, + 0x2329, 0x232A, 5, 0x232B, 0x23E8, 3, 0x23E9, 0x23EC, 5, 0x23ED, 0x23EF, 3, + 0x23F0, 0x23F0, 5, 0x23F1, 0x23F2, 3, 0x23F3, 0x23F3, 5, 0x23F4, 0x245F, 3, + 0x2460, 0x24E9, 0, 0x24EA, 0x24EA, 3, 0x24EB, 0x254B, 0, 0x254C, 0x254F, 3, + 0x2550, 0x2573, 0, 0x2574, 0x257F, 3, 0x2580, 0x258F, 0, 0x2590, 0x2591, 3, + 0x2592, 0x2595, 0, 0x2596, 0x259F, 3, 0x25A0, 0x25A1, 0, 0x25A2, 0x25A2, 3, + 0x25A3, 0x25A9, 0, 0x25AA, 0x25B1, 3, 0x25B2, 0x25B3, 0, 0x25B4, 0x25B5, 3, + 0x25B6, 0x25B7, 0, 0x25B8, 0x25BB, 3, 0x25BC, 0x25BD, 0, 0x25BE, 0x25BF, 3, + 0x25C0, 0x25C1, 0, 0x25C2, 0x25C5, 3, 0x25C6, 0x25C8, 0, 0x25C9, 0x25CA, 3, + 0x25CB, 0x25CB, 0, 0x25CC, 0x25CD, 3, 0x25CE, 0x25D1, 0, 0x25D2, 0x25E1, 3, + 0x25E2, 0x25E5, 0, 0x25E6, 0x25EE, 3, 0x25EF, 0x25EF, 0, 0x25F0, 0x25FC, 3, + 0x25FD, 0x25FE, 5, 0x25FF, 0x2604, 3, 0x2605, 0x2606, 0, 0x2607, 0x2608, 3, + 0x2609, 0x2609, 0, 0x260A, 0x260D, 3, 0x260E, 0x260F, 0, 0x2610, 0x2613, 3, + 0x2614, 0x2615, 5, 0x2616, 0x261B, 3, 0x261C, 0x261C, 0, 0x261D, 0x261D, 3, + 0x261E, 0x261E, 0, 0x261F, 0x262F, 3, 0x2630, 0x2637, 5, 0x2638, 0x263F, 3, + 0x2640, 0x2640, 0, 0x2641, 0x2641, 3, 0x2642, 0x2642, 0, 0x2643, 0x2647, 3, + 0x2648, 0x2653, 5, 0x2654, 0x265F, 3, 0x2660, 0x2661, 0, 0x2662, 0x2662, 3, + 0x2663, 0x2665, 0, 0x2666, 0x2666, 3, 0x2667, 0x266A, 0, 0x266B, 0x266B, 3, + 0x266C, 0x266D, 0, 0x266E, 0x266E, 3, 0x266F, 0x266F, 0, 0x2670, 0x267E, 3, + 0x267F, 0x267F, 5, 0x2680, 0x2689, 3, 0x268A, 0x268F, 5, 0x2690, 0x2692, 3, + 0x2693, 0x2693, 5, 0x2694, 0x269D, 3, 0x269E, 0x269F, 0, 0x26A0, 0x26A0, 3, + 0x26A1, 0x26A1, 5, 0x26A2, 0x26A9, 3, 0x26AA, 0x26AB, 5, 0x26AC, 0x26BC, 3, + 0x26BD, 0x26BE, 5, 0x26BF, 0x26BF, 0, 0x26C0, 0x26C3, 3, 0x26C4, 0x26C5, 5, + 0x26C6, 0x26CD, 0, 0x26CE, 0x26CE, 5, 0x26CF, 0x26D3, 0, 0x26D4, 0x26D4, 5, + 0x26D5, 0x26E1, 0, 0x26E2, 0x26E2, 3, 0x26E3, 0x26E3, 0, 0x26E4, 0x26E7, 3, + 0x26E8, 0x26E9, 0, 0x26EA, 0x26EA, 5, 0x26EB, 0x26F1, 0, 0x26F2, 0x26F3, 5, + 0x26F4, 0x26F4, 0, 0x26F5, 0x26F5, 5, 0x26F6, 0x26F9, 0, 0x26FA, 0x26FA, 5, + 0x26FB, 0x26FC, 0, 0x26FD, 0x26FD, 5, 0x26FE, 0x26FF, 0, 0x2700, 0x2704, 3, + 0x2705, 0x2705, 5, 0x2706, 0x2709, 3, 0x270A, 0x270B, 5, 0x270C, 0x2727, 3, + 0x2728, 0x2728, 5, 0x2729, 0x273C, 3, 0x273D, 0x273D, 0, 0x273E, 0x274B, 3, + 0x274C, 0x274C, 5, 0x274D, 0x274D, 3, 0x274E, 0x274E, 5, 0x274F, 0x2752, 3, + 0x2753, 0x2755, 5, 0x2756, 0x2756, 3, 0x2757, 0x2757, 5, 0x2758, 0x2775, 3, + 0x2776, 0x277F, 0, 0x2780, 0x2794, 3, 0x2795, 0x2797, 5, 0x2798, 0x27AF, 3, + 0x27B0, 0x27B0, 5, 0x27B1, 0x27BE, 3, 0x27BF, 0x27BF, 5, 0x27C0, 0x27E5, 3, + 0x27E6, 0x27ED, 4, 0x27EE, 0x2984, 3, 0x2985, 0x2986, 4, 0x2987, 0x2B1A, 3, + 0x2B1B, 0x2B1C, 5, 0x2B1D, 0x2B4F, 3, 0x2B50, 0x2B50, 5, 0x2B51, 0x2B54, 3, + 0x2B55, 0x2B55, 5, 0x2B56, 0x2B59, 0, 0x2B5A, 0x2E7F, 3, 0x2E80, 0x2E99, 5, + 0x2E9A, 0x2E9A, 3, 0x2E9B, 0x2EF3, 5, 0x2EF4, 0x2EFF, 3, 0x2F00, 0x2FD5, 5, + 0x2FD6, 0x2FEF, 3, 0x2FF0, 0x2FFF, 5, 0x3000, 0x3000, 1, 0x3001, 0x303E, 5, + 0x303F, 0x3040, 3, 0x3041, 0x3096, 5, 0x3097, 0x3098, 3, 0x3099, 0x30FF, 5, + 0x3100, 0x3104, 3, 0x3105, 0x312F, 5, 0x3130, 0x3130, 3, 0x3131, 0x318E, 5, + 0x318F, 0x318F, 3, 0x3190, 0x31E5, 5, 0x31E6, 0x31EE, 3, 0x31EF, 0x321E, 5, + 0x321F, 0x321F, 3, 0x3220, 0x3247, 5, 0x3248, 0x324F, 0, 0x3250, 0xA48C, 5, + 0xA48D, 0xA48F, 3, 0xA490, 0xA4C6, 5, 0xA4C7, 0xA95F, 3, 0xA960, 0xA97C, 5, + 0xA97D, 0xABFF, 3, 0xAC00, 0xD7A3, 5, 0xD7A4, 0xDFFF, 3, 0xE000, 0xF8FF, 0, + 0xF900, 0xFAFF, 5, 0xFB00, 0xFDFF, 3, 0xFE00, 0xFE0F, 0, 0xFE10, 0xFE19, 5, + 0xFE1A, 0xFE2F, 3, 0xFE30, 0xFE52, 5, 0xFE53, 0xFE53, 3, 0xFE54, 0xFE66, 5, + 0xFE67, 0xFE67, 3, 0xFE68, 0xFE6B, 5, 0xFE6C, 0xFF00, 3, 0xFF01, 0xFF60, 1, + 0xFF61, 0xFFBE, 2, 0xFFBF, 0xFFC1, 3, 0xFFC2, 0xFFC7, 2, 0xFFC8, 0xFFC9, 3, + 0xFFCA, 0xFFCF, 2, 0xFFD0, 0xFFD1, 3, 0xFFD2, 0xFFD7, 2, 0xFFD8, 0xFFD9, 3, + 0xFFDA, 0xFFDC, 2, 0xFFDD, 0xFFDF, 3, 0xFFE0, 0xFFE6, 1, 0xFFE7, 0xFFE7, 3, + 0xFFE8, 0xFFEE, 2, 0xFFEF, 0xFFFC, 3, 0xFFFD, 0xFFFD, 0, 0xFFFE, 0x16FDF, 3, + 0x16FE0, 0x16FE4, 5, 0x16FE5, 0x16FEF, 3, 0x16FF0, 0x16FF6, 5, 0x16FF7, 0x16FFF, 3, + 0x17000, 0x18CD5, 5, 0x18CD6, 0x18CFE, 3, 0x18CFF, 0x18D1E, 5, 0x18D1F, 0x18D7F, 3, + 0x18D80, 0x18DF2, 5, 0x18DF3, 0x1AFEF, 3, 0x1AFF0, 0x1AFF3, 5, 0x1AFF4, 0x1AFF4, 3, + 0x1AFF5, 0x1AFFB, 5, 0x1AFFC, 0x1AFFC, 3, 0x1AFFD, 0x1AFFE, 5, 0x1AFFF, 0x1AFFF, 3, + 0x1B000, 0x1B122, 5, 0x1B123, 0x1B131, 3, 0x1B132, 0x1B132, 5, 0x1B133, 0x1B14F, 3, + 0x1B150, 0x1B152, 5, 0x1B153, 0x1B154, 3, 0x1B155, 0x1B155, 5, 0x1B156, 0x1B163, 3, + 0x1B164, 0x1B167, 5, 0x1B168, 0x1B16F, 3, 0x1B170, 0x1B2FB, 5, 0x1B2FC, 0x1D2FF, 3, + 0x1D300, 0x1D356, 5, 0x1D357, 0x1D35F, 3, 0x1D360, 0x1D376, 5, 0x1D377, 0x1F003, 3, + 0x1F004, 0x1F004, 5, 0x1F005, 0x1F0CE, 3, 0x1F0CF, 0x1F0CF, 5, 0x1F0D0, 0x1F0FF, 3, + 0x1F100, 0x1F10A, 0, 0x1F10B, 0x1F10F, 3, 0x1F110, 0x1F12D, 0, 0x1F12E, 0x1F12F, 3, + 0x1F130, 0x1F169, 0, 0x1F16A, 0x1F16F, 3, 0x1F170, 0x1F18D, 0, 0x1F18E, 0x1F18E, 5, + 0x1F18F, 0x1F190, 0, 0x1F191, 0x1F19A, 5, 0x1F19B, 0x1F1AC, 0, 0x1F1AD, 0x1F1FF, 3, + 0x1F200, 0x1F202, 5, 0x1F203, 0x1F20F, 3, 0x1F210, 0x1F23B, 5, 0x1F23C, 0x1F23F, 3, + 0x1F240, 0x1F248, 5, 0x1F249, 0x1F24F, 3, 0x1F250, 0x1F251, 5, 0x1F252, 0x1F25F, 3, + 0x1F260, 0x1F265, 5, 0x1F266, 0x1F2FF, 3, 0x1F300, 0x1F320, 5, 0x1F321, 0x1F32C, 3, + 0x1F32D, 0x1F335, 5, 0x1F336, 0x1F336, 3, 0x1F337, 0x1F37C, 5, 0x1F37D, 0x1F37D, 3, + 0x1F37E, 0x1F393, 5, 0x1F394, 0x1F39F, 3, 0x1F3A0, 0x1F3CA, 5, 0x1F3CB, 0x1F3CE, 3, + 0x1F3CF, 0x1F3D3, 5, 0x1F3D4, 0x1F3DF, 3, 0x1F3E0, 0x1F3F0, 5, 0x1F3F1, 0x1F3F3, 3, + 0x1F3F4, 0x1F3F4, 5, 0x1F3F5, 0x1F3F7, 3, 0x1F3F8, 0x1F43E, 5, 0x1F43F, 0x1F43F, 3, + 0x1F440, 0x1F440, 5, 0x1F441, 0x1F441, 3, 0x1F442, 0x1F4FC, 5, 0x1F4FD, 0x1F4FE, 3, + 0x1F4FF, 0x1F53D, 5, 0x1F53E, 0x1F54A, 3, 0x1F54B, 0x1F54E, 5, 0x1F54F, 0x1F54F, 3, + 0x1F550, 0x1F567, 5, 0x1F568, 0x1F579, 3, 0x1F57A, 0x1F57A, 5, 0x1F57B, 0x1F594, 3, + 0x1F595, 0x1F596, 5, 0x1F597, 0x1F5A3, 3, 0x1F5A4, 0x1F5A4, 5, 0x1F5A5, 0x1F5FA, 3, + 0x1F5FB, 0x1F64F, 5, 0x1F650, 0x1F67F, 3, 0x1F680, 0x1F6C5, 5, 0x1F6C6, 0x1F6CB, 3, + 0x1F6CC, 0x1F6CC, 5, 0x1F6CD, 0x1F6CF, 3, 0x1F6D0, 0x1F6D2, 5, 0x1F6D3, 0x1F6D4, 3, + 0x1F6D5, 0x1F6D8, 5, 0x1F6D9, 0x1F6DB, 3, 0x1F6DC, 0x1F6DF, 5, 0x1F6E0, 0x1F6EA, 3, + 0x1F6EB, 0x1F6EC, 5, 0x1F6ED, 0x1F6F3, 3, 0x1F6F4, 0x1F6FC, 5, 0x1F6FD, 0x1F7DF, 3, + 0x1F7E0, 0x1F7EB, 5, 0x1F7EC, 0x1F7EF, 3, 0x1F7F0, 0x1F7F0, 5, 0x1F7F1, 0x1F90B, 3, + 0x1F90C, 0x1F93A, 5, 0x1F93B, 0x1F93B, 3, 0x1F93C, 0x1F945, 5, 0x1F946, 0x1F946, 3, + 0x1F947, 0x1F9FF, 5, 0x1FA00, 0x1FA6F, 3, 0x1FA70, 0x1FA7C, 5, 0x1FA7D, 0x1FA7F, 3, + 0x1FA80, 0x1FA8A, 5, 0x1FA8B, 0x1FA8D, 3, 0x1FA8E, 0x1FAC6, 5, 0x1FAC7, 0x1FAC7, 3, + 0x1FAC8, 0x1FAC8, 5, 0x1FAC9, 0x1FACC, 3, 0x1FACD, 0x1FADC, 5, 0x1FADD, 0x1FADE, 3, + 0x1FADF, 0x1FAEA, 5, 0x1FAEB, 0x1FAEE, 3, 0x1FAEF, 0x1FAF8, 5, 0x1FAF9, 0x1FFFF, 3, + 0x20000, 0x2FFFD, 5, 0x2FFFE, 0x2FFFF, 3, 0x30000, 0x3FFFD, 5, 0x3FFFE, 0xE00FF, 3, + 0xE0100, 0xE01EF, 0, 0xE01F0, 0xEFFFF, 3, 0xF0000, 0xFFFFD, 0, 0xFFFFE, 0xFFFFF, 3, + 0x100000, 0x10FFFD, 0, 0x10FFFE, 0x10FFFF, 3, + }; + + private static final UnicodeSet[] VALUE_SETS = buildValueSets(); + + static boolean isPropertyAlias(String alias) { + String loose = looseName(alias); + if (loose == null) return false; + for (String candidate : PROPERTY_ALIASES) { + if (candidate.equals(loose)) return true; + } + return false; + } + + static UnicodeSet valueSet(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : VALUE_SETS[index]; + } + + static String shortValue(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : SHORT_VALUES[index]; + } + + static String canonicalValue(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : LONG_VALUES[index]; + } + + static String[] canonicalValues() { + return LONG_VALUES.clone(); + } + + private static int valueIndex(String alias) { + String loose = looseName(alias); + if (loose == null) return -1; + for (int i = 0; i < VALUE_ALIASES.length; i++) { + if (VALUE_ALIASES[i].equals(loose)) return VALUE_ALIAS_INDEX[i]; + } + return -1; + } + + private static String looseName(String name) { + if (name == null) return null; + StringBuilder loose = new StringBuilder(name.length()); + for (int i = 0; i < name.length(); i++) { + char character = name.charAt(i); + if (character == '_' || character == '-' || Character.isWhitespace(character)) continue; + loose.append(Character.toLowerCase(character)); + } + return loose.toString(); + } + + private static UnicodeSet[] buildValueSets() { + UnicodeSet[] sets = new UnicodeSet[LONG_VALUES.length]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + for (int i = 0; i < RANGES.length; i += 3) { + sets[RANGES[i + 2]].add(RANGES[i], RANGES[i + 1]); + } + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + + private PerlUnicodeEastAsianWidthData() { + } +} diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index d6cf6dfbe6..0641292fa5 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -985,6 +985,18 @@ && isCanonicalCombiningClassProperty(alias.substring(0, assignment))) { } return PERL_DECOMPOSITION_TYPE_SETS[value]; } + if (assignment > 0 && assignment < alias.length() - 1 + && PerlUnicodeEastAsianWidthData.isPropertyAlias( + alias.substring(0, assignment))) { + UnicodeSet eastAsianWidth = PerlUnicodeEastAsianWidthData.valueSet( + alias.substring(assignment + 1)); + if (eastAsianWidth == null) { + throw new IllegalArgumentException( + "Unsupported East_Asian_Width value: " + + alias.substring(assignment + 1).trim()); + } + return eastAsianWidth; + } if (assignment > 0 && assignment < alias.length() - 1) { Boolean value = perlBooleanPropertyValue(alias.substring(assignment + 1)); if (value != null) { @@ -1310,8 +1322,12 @@ static String unicodeSetToJavaPattern(UnicodeSet set) { } private static void appendJavaPatternChar(StringBuilder sb, int codePoint) { - if (codePoint >= 0x10000) { - // Use \x{XXXX} for supplementary characters to avoid surrogate pair issues + if (codePoint >= 0x10000 + || (codePoint >= Character.MIN_SURROGATE + && codePoint <= Character.MAX_SURROGATE)) { + // Use \x{XXXX} for supplementary characters and surrogate code + // points. Literal unpaired surrogates cannot be encoded losslessly + // for Joni and can turn an otherwise valid range into an empty one. sb.append(String.format("\\x{%X}", codePoint)); } else { // Escape special regex metacharacters inside character classes diff --git a/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeEastAsianWidthDataTest.java b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeEastAsianWidthDataTest.java new file mode 100644 index 0000000000..bcb764c0ab --- /dev/null +++ b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeEastAsianWidthDataTest.java @@ -0,0 +1,99 @@ +package org.perlonjava.runtime.regex; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import com.ibm.icu.text.UnicodeSet; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; + +@Tag("unit") +class PerlUnicodeEastAsianWidthDataTest { + private static final String[][] VALUES = { + {"A", "Ambiguous", "00A1"}, + {"F", "Fullwidth", "FF01"}, + {"H", "Halfwidth", "FF61"}, + {"N", "Neutral", "0000"}, + {"Na", "Narrow", "0020"}, + {"W", "Wide", "1100"}, + }; + + @Test + void usesChecksumPinnedPerl544Unicode17Sources() { + assertEquals("17.0.0", PerlUnicodeEastAsianWidthData.UNICODE_VERSION); + assertEquals("0b5523a2217cb318d20b329a05d31eec5af5686ba09d263b85bb75a28989a3a8", + PerlUnicodeEastAsianWidthData.DEAST_ASIAN_WIDTH_SHA256); + assertEquals("670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01", + PerlUnicodeEastAsianWidthData.PROP_VALUE_ALIASES_SHA256); + assertEquals("4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb", + PerlUnicodeEastAsianWidthData.PROPERTY_ALIASES_SHA256); + } + + @Test + void acceptsPinnedPropertyAndLooseValueAliases() { + assertTrue(PerlUnicodeEastAsianWidthData.isPropertyAlias("ea")); + assertTrue(PerlUnicodeEastAsianWidthData.isPropertyAlias("East Asian Width")); + assertTrue(PerlUnicodeEastAsianWidthData.isPropertyAlias("EAST-ASIAN_WIDTH")); + assertFalse(PerlUnicodeEastAsianWidthData.isPropertyAlias("Emoji Width")); + + UnicodeSet ambiguous = PerlUnicodeEastAsianWidthData.valueSet("A"); + assertSame(ambiguous, PerlUnicodeEastAsianWidthData.valueSet("Ambiguous")); + assertSame(ambiguous, PerlUnicodeEastAsianWidthData.valueSet("a-m_b i g u o u s")); + assertEquals("A", PerlUnicodeEastAsianWidthData.shortValue("ambiguous")); + assertEquals("Ambiguous", PerlUnicodeEastAsianWidthData.canonicalValue("a")); + assertNull(PerlUnicodeEastAsianWidthData.valueSet("not a width")); + assertNull(PerlUnicodeEastAsianWidthData.canonicalValue(null)); + } + + @Test + void retainsEveryPinnedValue() { + assertEquals(6, PerlUnicodeEastAsianWidthData.canonicalValues().length); + for (String[] value : VALUES) { + UnicodeSet shortSet = PerlUnicodeEastAsianWidthData.valueSet(value[0]); + assertNotNull(shortSet, value[0]); + assertTrue(shortSet.isFrozen(), value[0]); + assertSame(shortSet, PerlUnicodeEastAsianWidthData.valueSet(value[1]), value[0]); + assertTrue(shortSet.contains(Integer.parseInt(value[2], 16)), value[0]); + } + } + + @Test + void appliesEveryOrderedMissingDefaultBeforeExplicitRanges() { + assertTrue(PerlUnicodeEastAsianWidthData.valueSet("W").contains(0x3FF0)); + assertTrue(PerlUnicodeEastAsianWidthData.valueSet("W").contains(0x2FA20)); + assertTrue(PerlUnicodeEastAsianWidthData.valueSet("W").contains(0x3FFFD)); + assertTrue(PerlUnicodeEastAsianWidthData.valueSet("N").contains(0x0378)); + assertTrue(PerlUnicodeEastAsianWidthData.valueSet("N").contains(0x10FFFF)); + assertTrue(PerlUnicodeEastAsianWidthData.valueSet("F").contains(0xFF01)); + } + + @Test + void formsOneDisjointPartitionOfAllUnicodeCodePoints() { + String[] values = PerlUnicodeEastAsianWidthData.canonicalValues(); + UnicodeSet union = new UnicodeSet(); + for (int i = 0; i < values.length; i++) { + UnicodeSet current = PerlUnicodeEastAsianWidthData.valueSet(values[i]); + union.addAll(current); + for (int j = i + 1; j < values.length; j++) { + UnicodeSet overlap = new UnicodeSet(current) + .retainAll(PerlUnicodeEastAsianWidthData.valueSet(values[j])); + assertTrue(overlap.isEmpty(), values[i] + " overlaps " + values[j]); + } + } + assertEquals(0x110000, union.size()); + assertTrue(union.contains(0, 0x10FFFF)); + } + + @Test + void rendersNeutralSurrogateRangeWithoutLiteralSurrogates() { + String pattern = UnicodeResolver.unicodeSetToJavaPattern( + PerlUnicodeEastAsianWidthData.valueSet("Neutral")); + + assertTrue(pattern.contains("\\x{DFFF}")); + assertFalse(pattern.matches(".*[\\uD800-\\uDFFF].*")); + } +} diff --git a/src/test/resources/unit/regex/unicode_east_asian_width_pinned.t b/src/test/resources/unit/regex/unicode_east_asian_width_pinned.t new file mode 100644 index 0000000000..f4523ef413 --- /dev/null +++ b/src/test/resources/unit/regex/unicode_east_asian_width_pinned.t @@ -0,0 +1,50 @@ +use strict; +use warnings; +use Test::More; + +my $east_asian_width_probe = eval q{qr/\p{ea=W}/}; +if (!defined $east_asian_width_probe) { + plan skip_all => 'pinned East_Asian_Width data awaits resolver integration'; +} + +my @values = ( + ['A', 'Ambiguous', 0x00A1], + ['F', 'Fullwidth', 0xFF01], + ['H', 'Halfwidth', 0xFF61], + ['N', 'Neutral', 0x0000], + ['Na', 'Narrow', 0x0020], + ['W', 'Wide', 0x1100], +); + +for my $value (@values) { + my ($short, $long, $code_point) = @$value; + my $character = chr($code_point); + my $short_pattern = eval "qr/\\p{ea=$short}/"; + ok(defined $short_pattern, "short value $short compiles") + or diag($@); + like($character, $short_pattern, "short value $short matches representative"); + + my $long_pattern = eval "qr/\\p{East_Asian_Width=$long}/"; + ok(defined $long_pattern, "long value $long compiles") + or diag($@); + like($character, $long_pattern, "long value $long matches representative"); +} + +my $loose = eval q{qr/\p{east asian width = ambiguous}/}; +ok(defined $loose, 'loose property and value aliases compile') or diag($@); +like(chr(0x00A1), $loose, 'loose property and value aliases match'); + +my @defaults = ( + [0x3FF0, 'W', 'Extension A missing default is Wide'], + [0x2FA20, 'W', 'supplementary CJK missing default is Wide'], + [0x3FFFD, 'W', 'plane 3 CJK missing default is Wide'], + [0x0378, 'N', 'general missing default is Neutral'], + [0x10FFFF, 'N', 'final code point uses the general Neutral default'], +); +for my $default (@defaults) { + my ($code_point, $value, $description) = @$default; + my $pattern = eval "qr/\\p{ea=$value}/"; + like(chr($code_point), $pattern, $description); +} + +done_testing(); From eb999477d2501210b123e6f79bb9c102e36f21f7 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:53 +0200 Subject: [PATCH 32/60] feat(regex): add pinned Numeric_Value parity (#1037) Squashed integration of PR #1037 at 8bfd23956. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 36 +- ...enerate_perl_unicode_numeric_value_data.pl | 409 ++++++++++++ docs/reference/feature-matrix.md | 4 +- .../regex/PerlUnicodeNumericValueData.java | 595 ++++++++++++++++++ .../runtime/regex/UnicodeResolver.java | 153 +++++ .../PerlUnicodeNumericValueDataTest.java | 149 +++++ .../unit/regex/unicode_numeric_value_data.t | 76 +++ 7 files changed, 1411 insertions(+), 11 deletions(-) create mode 100644 dev/tools/generate_perl_unicode_numeric_value_data.pl create mode 100644 src/main/java/org/perlonjava/runtime/regex/PerlUnicodeNumericValueData.java create mode 100644 src/test/java/org/perlonjava/runtime/regex/PerlUnicodeNumericValueDataTest.java create mode 100644 src/test/resources/unit/regex/unicode_numeric_value_data.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 97a43b0405..3f855e0db6 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,7 +224,7 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1035, stacked +The published integration stack is preserved through draft PR #1036, stacked on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the @@ -237,9 +237,11 @@ binary `ASCII_Hex_Digit` values, pinned General_Category sets, and exact native line boundaries. PR #1031 adds pinned Canonical_Combining_Class sets and valid empty-property rendering. PR #1032 integrates native numeric escapes through U+10FFFF; PR #1033 adds pinned Bidi_Class sets; PR #1034 integrates native -vertical-whitespace escapes; PR #1035 adds Decomposition_Type. The current WIP -integrates East_Asian_Width; Numeric_Value and Joining_Group are ready, while -Block and Script/Script_Extensions data slices advance independently. +vertical-whitespace escapes; PR #1035 adds Decomposition_Type and PR #1036 +adds East_Asian_Width. The current WIP integrates Numeric_Value; independently +validated Joining_Group, Block, Script/Script_Extensions, and break-property +value data are ready, while generic and specialized binary-property slices +advance independently. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -348,6 +350,17 @@ and interpreter; protected boundary smoke remains 169/169 per backend. Chunk with zero numbered regressions, raising current generated evidence to 342,872/407,367. +`Numeric_Value`/`nv` assignments now resolve all 144 exact rational values and +the `NaN` complement from pinned Perl 5.44 Unicode 17 data. Integer, decimal, +exponent, reduced-rational, loose, wildcard, and exact case-sensitive `Is` +forms follow Perl's generated keyword aliases and binary-NV canonicalization, +including four-significant-digit decimal spellings without heuristic tolerance. +The focused oracle passes 50/50 +on system Perl, JVM, and interpreter; protected boundary smoke remains 169/169 +per backend. Chunks 02–03 gain 13,976 assertions with zero numbered regressions +and exact JVM/interpreter success sets, raising current generated evidence to +356,848/407,367. + Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while ordinary and substitution callbacks execute, without misclassifying escaped or @@ -608,6 +621,11 @@ is retained for now. surrogate-range rendering. The focused oracle passes 31/31 and chunk 01 gains 216 assertions to 35,108/41,843 on both backends with zero numbered regressions. + - [x] Generated and integrated pinned Perl 5.44 Unicode 17.0 Numeric_Value + sets for all 144 rationals plus NaN, including exact rational reduction, + generated decimal keyword aliases, loose forms, and wildcard policy. The + focused oracle passes 50/50; chunks 02–03 gain 13,976 assertions with zero + numbered regressions and exact backend identity. - [x] Integrated native Perl `\v`/`\V` dispatch inside and outside character classes (`1eff1db97`, integrated as `6328935cd`). The focused oracle passes 92/92 and unchanged `reg_posixcc.t` passes 2,560/2,560 on both backends. @@ -653,15 +671,15 @@ is retained for now. ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1035. Publish the validated - East_Asian_Width integration in a separate focused WIP PR, preserving the +1. Land review-ready PR #1024 and draft PRs #1025–#1036. Publish the validated + Numeric_Value integration in a separate focused WIP PR, preserving the independently reviewed data commit. 2. Preserve the now-complete native Joni boundary corpus at 239,866/239,866: sentence chunk 05, line chunks 06–09, and word chunk 10 must remain exact on JVM and interpreter while property and parser work continues. -3. Integrate the independently generated Numeric_Value and Joining_Group - slices, then continue the measured residual order: binary - property values, Block, Script/Script_Extensions, and break-property values. +3. Integrate the independently generated Joining_Group, Block, + Script/Script_Extensions, and break-property value slices, then the generic + and specialized binary-property families currently advancing in parallel. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. Keep native `\v`/`\V` exact at 2,560/2,560 in `reg_posixcc.t`. diff --git a/dev/tools/generate_perl_unicode_numeric_value_data.pl b/dev/tools/generate_perl_unicode_numeric_value_data.pl new file mode 100644 index 0000000000..5d4b0100f7 --- /dev/null +++ b/dev/tools/generate_perl_unicode_numeric_value_data.pl @@ -0,0 +1,409 @@ +#!/usr/bin/env perl +use strict; +use warnings; +use Config; +use Digest::SHA qw(sha256_hex); +use File::Spec; +use FindBin; + +binmode STDOUT, ':raw'; + +my $expected_version = '17.0.0'; +my $unicore = File::Spec->catdir($FindBin::Bin, '..', '..', 'perl5', 'lib', 'unicore'); +my %sources = ( + Version => [File::Spec->catfile($unicore, 'version'), + '8c30575264b2772c7a69c5bb6069a28f0e0a7a0df735871bde2d99ee674316ac'], + Numeric_Value => [File::Spec->catfile($unicore, 'extracted', 'DNumValues.txt'), + '139b976bdc288be01c80f018523da769cf2845109b5a7f0f8a432db64bfedcfa'], + Property_Aliases => [File::Spec->catfile($unicore, 'PropertyAliases.txt'), + '4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb'], + Property_Value_Aliases => [File::Spec->catfile($unicore, 'PropValueAliases.txt'), + '670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01'], +); + +die "Numeric_Value generation requires 64-bit Perl integers\n" + unless $Config{ivsize} >= 8; + +sub source_text { + my ($name) = @_; + my ($path, $expected_hash) = @{$sources{$name}}; + open my $input, '<:raw', $path or die "Cannot read $path: $!\n"; + local $/; + my $text = <$input>; + close $input or die "Cannot close $path: $!\n"; + my $actual_hash = sha256_hex($text); + die "$path SHA-256 mismatch: expected $expected_hash, found $actual_hash\n" + unless $actual_hash eq $expected_hash; + return ($path, $text); +} + +sub trim { + my ($text) = @_; + $text =~ s/^\s+|\s+$//g; + return $text; +} + +sub loose { + my ($text) = @_; + $text = lc $text; + $text =~ s/[\x09-\x0d _-]+//g; + return $text; +} + +sub range_from_text { + my ($range) = @_; + my ($first, $last) = split /\.\./, $range; + return (hex($first), hex(defined $last ? $last : $first)); +} + +sub verify_unicode_notice { + my ($path, $text) = @_; + die "$path does not preserve the Unicode copyright notice\n" + unless $text =~ /^# © 2025 Unicode®, Inc\.$/m; + die "$path does not preserve the Unicode terms notice\n" + unless $text =~ m{^# For terms of use and license, see https://www\.unicode\.org/terms_of_use\.html$}m; +} + +sub gcd { + my ($left, $right) = @_; + $left = -$left if $left < 0; + while ($right != 0) { + ($left, $right) = ($right, $left % $right); + } + return $left; +} + +my ($version_path, $version_text) = source_text('Version'); +$version_text =~ s/\s+\z//; +die "Expected Unicode $expected_version, found '$version_text' in $version_path\n" + unless $version_text eq $expected_version; + +my ($property_path, $property_text) = source_text('Property_Aliases'); +die "$property_path is not pinned Unicode $expected_version data\n" + unless $property_text =~ /^# PropertyAliases-\Q$expected_version\E\.txt$/m; +verify_unicode_notice($property_path, $property_text); +my @property_aliases; +for my $line (split /\n/, $property_text) { + $line =~ s/#.*//; + my @fields = map { trim($_) } split /;/, $line, -1; + next unless @fields >= 2 && $fields[0] eq 'nv'; + @property_aliases = grep { length } @fields; +} +die "Missing nv aliases in $property_path\n" unless @property_aliases; +die "Unexpected nv aliases in $property_path: @property_aliases\n" + unless join("\0", @property_aliases) eq join("\0", 'nv', 'Numeric_Value'); + +my ($value_path, $value_text) = source_text('Property_Value_Aliases'); +die "$value_path is not pinned Unicode $expected_version data\n" + unless $value_text =~ /^# PropertyValueAliases-\Q$expected_version\E\.txt$/m; +verify_unicode_notice($value_path, $value_text); +my @missing; +for my $line (split /\n/, $value_text) { + if ($line =~ /^\#\s*\@missing:\s*([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*; + \s*Numeric_Value\s*;\s*([A-Za-z0-9_]+)/x) { + my ($first, $last) = range_from_text($1); + push @missing, [$first, $last, $2]; + } +} +die "Expected one Numeric_Value \@missing rule\n" + unless @missing == 1 && $missing[0][0] == 0 && $missing[0][1] == 0x10ffff + && $missing[0][2] eq 'NaN'; + +my ($data_path, $data_text) = source_text('Numeric_Value'); +die "$data_path is not pinned Unicode $expected_version data\n" + unless $data_text =~ /^# DerivedNumericValues-\Q$expected_version\E\.txt$/m; +verify_unicode_notice($data_path, $data_text); + +my (@ranges, @values, %value_index, %decimal_value, %source_counts); +my ($record_count, $range_record_count, $explicit_count) = (0, 0, 0); +for my $line (split /\n/, $data_text) { + next if $line =~ /^\s*(?:#|$)/; + die "Malformed Numeric_Value record '$line'\n" + unless $line =~ /^([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*; + \s*([^;]+?)\s*;\s*;\s*(-?\d+(?:\/\d+)?)\s*(?:\#.*)?$/x; + my ($range_text, $decimal, $canonical) = ($1, trim($2), $3); + my ($first, $last) = range_from_text($range_text); + my ($numerator, $denominator) = $canonical =~ m{^(-?\d+)(?:/(\d+))?$}; + $denominator = 1 unless defined $denominator; + die "Invalid rational '$canonical'\n" + if $denominator <= 0 || gcd($numerator, $denominator) != 1; + die "Numeric value does not fit signed long: '$canonical'\n" + if $numerator < -9_223_372_036_854_775_807 + || $numerator > 9_223_372_036_854_775_807; + die "Conflicting decimal spelling '$decimal'\n" + if exists $decimal_value{$decimal} && $decimal_value{$decimal} ne $canonical; + $decimal_value{$decimal} = $canonical; + + if (!exists $value_index{$canonical}) { + $value_index{$canonical} = scalar @values; + my $perl_decimal_alias = sprintf '%.3e', $numerator / $denominator; + push @values, [ + $canonical, 0 + $numerator, 0 + $denominator, $perl_decimal_alias + ]; + } + push @ranges, [$first, $last, $value_index{$canonical}]; + $record_count++; + $range_record_count++ if $range_text =~ /\.\./; + $explicit_count += $last - $first + 1; + $source_counts{$canonical} += $last - $first + 1; +} + +die "Expected 1,980 Numeric_Value records, found $record_count\n" + unless $record_count == 1_980; +die "Expected 25 source range records, found $range_record_count\n" + unless $range_record_count == 25; +die "Expected 2,023 numeric code points, found $explicit_count\n" + unless $explicit_count == 2_023; +die "Expected 144 distinct numeric values, found " . scalar(@values) . "\n" + unless @values == 144; +die "Expected 144 distinct decimal spellings, found " . scalar(keys %decimal_value) . "\n" + unless keys(%decimal_value) == 144; +my %decimal_canonical = map { $decimal_value{$_} => 1 } keys %decimal_value; +die "Decimal spellings do not map one-to-one to the 144 exact values\n" + unless keys(%decimal_canonical) == 144; +my %perl_decimal_for = map { $_->[0] => $_->[3] } @values; +for my $expected ( + ['1/12', '8.333e-02'], ['1/64', '1.562e-02'], + ['1/7', '1.429e-01'], ['1/6', '1.667e-01'], ['3/64', '4.688e-02'], +) { + die "Unexpected Perl decimal alias for $expected->[0]: " + . ($perl_decimal_for{$expected->[0]} // '') . "\n" + unless ($perl_decimal_for{$expected->[0]} // '') eq $expected->[1]; +} + +@ranges = sort { $a->[0] <=> $b->[0] } @ranges; +for my $index (0 .. $#ranges) { + my $range = $ranges[$index]; + die sprintf("Invalid Numeric_Value range U+%04X..U+%04X\n", @$range[0, 1]) + if $range->[0] < 0 || $range->[1] > 0x10ffff || $range->[0] > $range->[1]; + die sprintf("Overlapping Numeric_Value ranges at U+%04X\n", $range->[0]) + if $index > 0 && $range->[0] <= $ranges[$index - 1][1]; +} + +my @coalesced; +for my $range (@ranges) { + if (@coalesced && $coalesced[-1][2] == $range->[2] + && $coalesced[-1][1] + 1 == $range->[0]) { + $coalesced[-1][1] = $range->[1]; + } else { + push @coalesced, [@$range]; + } +} +die "Expected 1,979 coalesced numeric ranges, found " . scalar(@coalesced) . "\n" + unless @coalesced == 1_979; +die "Expected 1,112,089 NaN code points\n" + unless 0x110000 - $explicit_count == 1_112_089; + +my @ranges_by_value = map { [] } @values; +for my $range (@coalesced) { + push @{$ranges_by_value[$range->[2]]}, [$range->[0], $range->[1]]; +} +for my $index (0 .. $#values) { + my $count = 0; + $count += $_->[1] - $_->[0] + 1 for @{$ranges_by_value[$index]}; + die "Cardinality changed for numeric value '$values[$index][0]'\n" + unless $count == $source_counts{$values[$index][0]}; +} + +my (@range_offsets, @range_endpoints); +for my $ranges_for_value (@ranges_by_value) { + push @range_offsets, scalar(@range_endpoints) / 2; + push @range_endpoints, @$_ for @$ranges_for_value; +} +push @range_offsets, scalar(@range_endpoints) / 2; + +print <<'HEADER'; +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; +import java.math.BigDecimal; +import java.math.MathContext; +import java.math.RoundingMode; + +/* + * Generated from Perl 5.44's pinned Unicode Character Database by + * dev/tools/generate_perl_unicode_numeric_value_data.pl. Do not edit manually. + * + * Unicode data source copyright: + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in + * the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + */ +final class PerlUnicodeNumericValueData { +HEADER + +print " static final String UNICODE_VERSION = \"$expected_version\";\n"; +print " static final short INVALID = -1;\n\n"; +print " private static final String[] CANONICAL_VALUES = {\n "; +print join(', ', map { qq{"$_->[0]"} } @values); +print "\n };\n\n"; +print " private static final String[] DECIMAL_ALIASES = {\n "; +print join(', ', map { qq{"$_->[3]"} } @values); +print "\n };\n\n"; +print " private static final long[] NUMERATORS = {\n "; +print join(', ', map { "$_->[1]L" } @values); +print "\n };\n\n"; +print " private static final short[] DENOMINATORS = {\n "; +print join(', ', map { $_->[2] } @values); +print "\n };\n\n"; +print " private static final int[] RANGE_OFFSETS = {\n"; +for (my $i = 0; $i < @range_offsets; $i += 16) { + my $end = $i + 15 < $#range_offsets ? $i + 15 : $#range_offsets; + print " ", join(', ', @range_offsets[$i .. $end]), ",\n"; +} +print " };\n\n private static final int[] RANGE_ENDPOINTS = {\n"; +for (my $i = 0; $i < @range_endpoints; $i += 10) { + my $end = $i + 9 < $#range_endpoints ? $i + 9 : $#range_endpoints; + print " ", join(', ', map { sprintf '0x%X', $range_endpoints[$_] } $i .. $end), ",\n"; +} +print <<'FOOTER'; + }; + + private static final MathContext PERL_NUMERIC_CONTEXT = + new MathContext(4, RoundingMode.HALF_EVEN); + private static final UnicodeSet[] SETS = buildSets(); + private static final BigDecimal[] DECIMALS = buildDecimals(); + private static final BigDecimal[] ALTERNATE_DECIMALS = buildAlternateDecimals(); + private static final UnicodeSet ASSIGNED = buildAssignedSet(); + private static final UnicodeSet NAN = new UnicodeSet(0, 0x10ffff) + .removeAll(ASSIGNED).freeze(); + + static int valueCount() { + return CANONICAL_VALUES.length; + } + + static String canonicalValue(int index) { + return CANONICAL_VALUES[index]; + } + + static long numerator(int index) { + return NUMERATORS[index]; + } + + static short valueForDecimal(BigDecimal decimal) { + double binaryValue = decimal.doubleValue(); + if (!Double.isFinite(binaryValue)) return INVALID; + BigDecimal canonical = new BigDecimal(binaryValue).round(PERL_NUMERIC_CONTEXT); + for (short index = 0; index < DECIMALS.length; index++) { + if (DENOMINATORS[index] != 1 + && (DECIMALS[index].compareTo(canonical) == 0 + || ALTERNATE_DECIMALS[index].compareTo(canonical) == 0)) { + return index; + } + } + return INVALID; + } + + static int denominator(int index) { + return DENOMINATORS[index]; + } + + static UnicodeSet set(int index) { + return SETS[index]; + } + + static UnicodeSet nanSet() { + return NAN; + } + + static UnicodeSet assignedSet() { + return ASSIGNED; + } + + static short valueForRational(long numerator, long denominator) { + if (denominator == 0 || numerator == Long.MIN_VALUE || denominator == Long.MIN_VALUE) { + return INVALID; + } + if (denominator < 0) { + numerator = -numerator; + denominator = -denominator; + } + long divisor = gcd(numerator, denominator); + numerator /= divisor; + denominator /= divisor; + for (short index = 0; index < CANONICAL_VALUES.length; index++) { + if (NUMERATORS[index] == numerator && DENOMINATORS[index] == denominator) { + return index; + } + } + return INVALID; + } + + static boolean isPropertyAlias(String alias) { + boolean hasIsPrefix = alias != null && alias.startsWith("Is"); + String normalized = loose(hasIsPrefix ? alias.substring(2) : alias); + switch (normalized) { + case "numericvalue": + case "nv": + return true; + default: + return false; + } + } + + private static UnicodeSet[] buildSets() { + UnicodeSet[] sets = new UnicodeSet[CANONICAL_VALUES.length]; + for (int value = 0; value < sets.length; value++) { + UnicodeSet set = new UnicodeSet(); + for (int range = RANGE_OFFSETS[value]; range < RANGE_OFFSETS[value + 1]; range++) { + set.add(RANGE_ENDPOINTS[range * 2], RANGE_ENDPOINTS[range * 2 + 1]); + } + sets[value] = set.freeze(); + } + return sets; + } + + private static BigDecimal[] buildDecimals() { + BigDecimal[] decimals = new BigDecimal[DECIMAL_ALIASES.length]; + for (int index = 0; index < decimals.length; index++) { + decimals[index] = new BigDecimal(DECIMAL_ALIASES[index]); + if (index > 0 && decimals[index - 1].compareTo(decimals[index]) >= 0) { + throw new IllegalStateException("Numeric_Value decimals are not sorted"); + } + } + return decimals; + } + + private static BigDecimal[] buildAlternateDecimals() { + MathContext alternateContext = new MathContext(4, RoundingMode.HALF_UP); + BigDecimal[] decimals = new BigDecimal[NUMERATORS.length]; + for (int index = 0; index < decimals.length; index++) { + decimals[index] = BigDecimal.valueOf(NUMERATORS[index]) + .divide(BigDecimal.valueOf(DENOMINATORS[index]), alternateContext); + } + return decimals; + } + + private static UnicodeSet buildAssignedSet() { + UnicodeSet assigned = new UnicodeSet(); + for (UnicodeSet set : SETS) assigned.addAll(set); + return assigned.freeze(); + } + + private static long gcd(long left, long right) { + left = Math.abs(left); + while (right != 0) { + long remainder = left % right; + left = right; + right = remainder; + } + return left; + } + + private static String loose(String alias) { + if (alias == null) return ""; + StringBuilder normalized = new StringBuilder(alias.length()); + for (int i = 0; i < alias.length(); i++) { + char character = alias.charAt(i); + if (character == '_' || character == '-' || character == ' ' + || (character >= '\t' && character <= '\r')) continue; + normalized.append(Character.toLowerCase(character)); + } + return normalized.toString(); + } + + private PerlUnicodeNumericValueData() { + } +} +FOOTER diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index 94c2389a2b..6d0de46109 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -379,7 +379,7 @@ my @copy = @{$z}; # ERROR - ✅ **Backreferences to Named Groups**: Using `\k` or `\g{name}` for backreferences to named groups is supported. - ✅ **Relative Backreferences**: Using `\g{-n}` for relative backreferences. - ✅ **Basic Unicode Properties**: Common `\p{...}` and `\P{...}` forms such as `\p{L}` execute through Joni. -- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, and East_Asian_Width assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, and reserved or composite values; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. +- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, and Numeric_Value assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, reserved or composite values, exact rationals, and Perl's generated decimal keyword aliases; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. - ✅ **Possessive Quantifiers**: Quantifiers like `*+`, `++`, `?+`, and `{n,m}+`, which disable backtracking, are supported. - ✅ **Atomic Grouping**: Use of `(?>...)` for atomic groups is supported. - ✅ **`\K` assertion**: Keep left — in `s///`, text before `\K` is preserved; match variables reflect only the portion after `\K`. @@ -398,7 +398,7 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 342,872/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, Numeric_Value, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 356,848/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. - ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. - ✅ **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter. - ✅ **Unicode Word Boundaries**: Native `\b{wb}`/`\B{wb}` implement WB1–WB16 and WB999 from reproducible Perl 5.44 Unicode 17.0 word-break and extended-pictographic tables. Authoritative chunk 10 passes 19,510/19,510 identically on JVM and interpreter. diff --git a/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeNumericValueData.java b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeNumericValueData.java new file mode 100644 index 0000000000..925df8a3f5 --- /dev/null +++ b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeNumericValueData.java @@ -0,0 +1,595 @@ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; +import java.math.BigDecimal; +import java.math.MathContext; +import java.math.RoundingMode; + +/* + * Generated from Perl 5.44's pinned Unicode Character Database by + * dev/tools/generate_perl_unicode_numeric_value_data.pl. Do not edit manually. + * + * Unicode data source copyright: + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in + * the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + */ +final class PerlUnicodeNumericValueData { + static final String UNICODE_VERSION = "17.0.0"; + static final short INVALID = -1; + + private static final String[] CANONICAL_VALUES = { + "-1/2", "0", "1/320", "1/160", "1/80", "1/64", "1/40", "1/32", "3/80", "3/64", "1/20", "1/16", "1/12", "1/10", "1/9", "1/8", "1/7", "3/20", "1/6", "3/16", "1/5", "1/4", "1/3", "3/8", "2/5", "5/12", "1/2", "7/12", "3/5", "5/8", "2/3", "3/4", "4/5", "5/6", "7/8", "11/12", "1", "3/2", "2", "5/2", "3", "7/2", "4", "9/2", "5", "11/2", "6", "13/2", "7", "15/2", "8", "17/2", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "40", "41", "42", "43", "44", "45", "46", "47", "48", "49", "50", "60", "70", "80", "90", "100", "200", "300", "400", "500", "600", "700", "800", "900", "1000", "2000", "3000", "4000", "5000", "6000", "7000", "8000", "9000", "10000", "20000", "30000", "40000", "50000", "60000", "70000", "80000", "90000", "100000", "200000", "216000", "300000", "400000", "432000", "500000", "600000", "700000", "800000", "900000", "1000000", "10000000", "20000000", "100000000", "1000000000", "10000000000", "1000000000000", "10000000000000000" + }; + + private static final String[] DECIMAL_ALIASES = { + "-5.000e-01", "0.000e+00", "3.125e-03", "6.250e-03", "1.250e-02", "1.562e-02", "2.500e-02", "3.125e-02", "3.750e-02", "4.688e-02", "5.000e-02", "6.250e-02", "8.333e-02", "1.000e-01", "1.111e-01", "1.250e-01", "1.429e-01", "1.500e-01", "1.667e-01", "1.875e-01", "2.000e-01", "2.500e-01", "3.333e-01", "3.750e-01", "4.000e-01", "4.167e-01", "5.000e-01", "5.833e-01", "6.000e-01", "6.250e-01", "6.667e-01", "7.500e-01", "8.000e-01", "8.333e-01", "8.750e-01", "9.167e-01", "1.000e+00", "1.500e+00", "2.000e+00", "2.500e+00", "3.000e+00", "3.500e+00", "4.000e+00", "4.500e+00", "5.000e+00", "5.500e+00", "6.000e+00", "6.500e+00", "7.000e+00", "7.500e+00", "8.000e+00", "8.500e+00", "9.000e+00", "1.000e+01", "1.100e+01", "1.200e+01", "1.300e+01", "1.400e+01", "1.500e+01", "1.600e+01", "1.700e+01", "1.800e+01", "1.900e+01", "2.000e+01", "2.100e+01", "2.200e+01", "2.300e+01", "2.400e+01", "2.500e+01", "2.600e+01", "2.700e+01", "2.800e+01", "2.900e+01", "3.000e+01", "3.100e+01", "3.200e+01", "3.300e+01", "3.400e+01", "3.500e+01", "3.600e+01", "3.700e+01", "3.800e+01", "3.900e+01", "4.000e+01", "4.100e+01", "4.200e+01", "4.300e+01", "4.400e+01", "4.500e+01", "4.600e+01", "4.700e+01", "4.800e+01", "4.900e+01", "5.000e+01", "6.000e+01", "7.000e+01", "8.000e+01", "9.000e+01", "1.000e+02", "2.000e+02", "3.000e+02", "4.000e+02", "5.000e+02", "6.000e+02", "7.000e+02", "8.000e+02", "9.000e+02", "1.000e+03", "2.000e+03", "3.000e+03", "4.000e+03", "5.000e+03", "6.000e+03", "7.000e+03", "8.000e+03", "9.000e+03", "1.000e+04", "2.000e+04", "3.000e+04", "4.000e+04", "5.000e+04", "6.000e+04", "7.000e+04", "8.000e+04", "9.000e+04", "1.000e+05", "2.000e+05", "2.160e+05", "3.000e+05", "4.000e+05", "4.320e+05", "5.000e+05", "6.000e+05", "7.000e+05", "8.000e+05", "9.000e+05", "1.000e+06", "1.000e+07", "2.000e+07", "1.000e+08", "1.000e+09", "1.000e+10", "1.000e+12", "1.000e+16" + }; + + private static final long[] NUMERATORS = { + -1L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 3L, 3L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 3L, 1L, 3L, 1L, 1L, 1L, 3L, 2L, 5L, 1L, 7L, 3L, 5L, 2L, 3L, 4L, 5L, 7L, 11L, 1L, 3L, 2L, 5L, 3L, 7L, 4L, 9L, 5L, 11L, 6L, 13L, 7L, 15L, 8L, 17L, 9L, 10L, 11L, 12L, 13L, 14L, 15L, 16L, 17L, 18L, 19L, 20L, 21L, 22L, 23L, 24L, 25L, 26L, 27L, 28L, 29L, 30L, 31L, 32L, 33L, 34L, 35L, 36L, 37L, 38L, 39L, 40L, 41L, 42L, 43L, 44L, 45L, 46L, 47L, 48L, 49L, 50L, 60L, 70L, 80L, 90L, 100L, 200L, 300L, 400L, 500L, 600L, 700L, 800L, 900L, 1000L, 2000L, 3000L, 4000L, 5000L, 6000L, 7000L, 8000L, 9000L, 10000L, 20000L, 30000L, 40000L, 50000L, 60000L, 70000L, 80000L, 90000L, 100000L, 200000L, 216000L, 300000L, 400000L, 432000L, 500000L, 600000L, 700000L, 800000L, 900000L, 1000000L, 10000000L, 20000000L, 100000000L, 1000000000L, 10000000000L, 1000000000000L, 10000000000000000L + }; + + private static final short[] DENOMINATORS = { + 2, 1, 320, 160, 80, 64, 40, 32, 80, 64, 20, 16, 12, 10, 9, 8, 7, 20, 6, 16, 5, 4, 3, 8, 5, 12, 2, 12, 5, 8, 3, 4, 5, 6, 8, 12, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 + }; + + private static final int[] RANGE_OFFSETS = { + 0, 1, 96, 98, 100, 101, 102, 104, 105, 107, 108, 110, 115, 116, 119, 120, + 127, 128, 130, 134, 139, 142, 155, 161, 162, 163, 164, 182, 183, 184, 185, 192, + 201, 202, 205, 206, 207, 361, 363, 518, 519, 666, 667, 806, 807, 948, 949, 1075, + 1076, 1200, 1201, 1321, 1322, 1446, 1505, 1514, 1523, 1530, 1537, 1544, 1552, 1560, 1568, 1576, + 1612, 1613, 1614, 1615, 1616, 1617, 1618, 1619, 1620, 1621, 1640, 1641, 1642, 1643, 1644, 1645, + 1646, 1647, 1648, 1649, 1667, 1668, 1669, 1670, 1671, 1672, 1673, 1674, 1675, 1676, 1702, 1715, + 1728, 1740, 1752, 1787, 1794, 1801, 1808, 1820, 1827, 1833, 1839, 1846, 1867, 1872, 1876, 1880, + 1888, 1892, 1896, 1900, 1904, 1917, 1921, 1925, 1929, 1936, 1940, 1944, 1948, 1952, 1957, 1959, + 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1974, 1975, 1976, 1978, + 1979, + }; + + private static final int[] RANGE_ENDPOINTS = { + 0xF33, 0xF33, 0x30, 0x30, 0x660, 0x660, 0x6F0, 0x6F0, 0x7C0, 0x7C0, + 0x966, 0x966, 0x9E6, 0x9E6, 0xA66, 0xA66, 0xAE6, 0xAE6, 0xB66, 0xB66, + 0xBE6, 0xBE6, 0xC66, 0xC66, 0xC78, 0xC78, 0xCE6, 0xCE6, 0xD66, 0xD66, + 0xDE6, 0xDE6, 0xE50, 0xE50, 0xED0, 0xED0, 0xF20, 0xF20, 0x1040, 0x1040, + 0x1090, 0x1090, 0x17E0, 0x17E0, 0x17F0, 0x17F0, 0x1810, 0x1810, 0x1946, 0x1946, + 0x19D0, 0x19D0, 0x1A80, 0x1A80, 0x1A90, 0x1A90, 0x1B50, 0x1B50, 0x1BB0, 0x1BB0, + 0x1C40, 0x1C40, 0x1C50, 0x1C50, 0x2070, 0x2070, 0x2080, 0x2080, 0x2189, 0x2189, + 0x24EA, 0x24EA, 0x24FF, 0x24FF, 0x3007, 0x3007, 0x6D1E, 0x6D1E, 0x96F6, 0x96F6, + 0xA620, 0xA620, 0xA6EF, 0xA6EF, 0xA8D0, 0xA8D0, 0xA900, 0xA900, 0xA9D0, 0xA9D0, + 0xA9F0, 0xA9F0, 0xAA50, 0xAA50, 0xABF0, 0xABF0, 0xF9B2, 0xF9B2, 0xFF10, 0xFF10, + 0x1018A, 0x1018A, 0x104A0, 0x104A0, 0x10D30, 0x10D30, 0x10D40, 0x10D40, 0x11066, 0x11066, + 0x110F0, 0x110F0, 0x11136, 0x11136, 0x111D0, 0x111D0, 0x112F0, 0x112F0, 0x11450, 0x11450, + 0x114D0, 0x114D0, 0x11650, 0x11650, 0x116C0, 0x116C0, 0x116D0, 0x116D0, 0x116DA, 0x116DA, + 0x11730, 0x11730, 0x118E0, 0x118E0, 0x11950, 0x11950, 0x11BF0, 0x11BF0, 0x11C50, 0x11C50, + 0x11D50, 0x11D50, 0x11DA0, 0x11DA0, 0x11DE0, 0x11DE0, 0x11F50, 0x11F50, 0x16130, 0x16130, + 0x16A60, 0x16A60, 0x16AC0, 0x16AC0, 0x16B50, 0x16B50, 0x16D70, 0x16D70, 0x16E80, 0x16E80, + 0x1CCF0, 0x1CCF0, 0x1D2C0, 0x1D2C0, 0x1D2E0, 0x1D2E0, 0x1D7CE, 0x1D7CE, 0x1D7D8, 0x1D7D8, + 0x1D7E2, 0x1D7E2, 0x1D7EC, 0x1D7EC, 0x1D7F6, 0x1D7F6, 0x1E140, 0x1E140, 0x1E2F0, 0x1E2F0, + 0x1E4F0, 0x1E4F0, 0x1E5F1, 0x1E5F1, 0x1E950, 0x1E950, 0x1F100, 0x1F101, 0x1F10B, 0x1F10C, + 0x1FBF0, 0x1FBF0, 0x11FC0, 0x11FC0, 0x11FD4, 0x11FD4, 0xD58, 0xD58, 0x11FC1, 0x11FC1, + 0x11FC2, 0x11FC2, 0x11FC3, 0x11FC3, 0xD59, 0xD59, 0x11FC4, 0x11FC4, 0x11FC5, 0x11FC5, + 0xD5A, 0xD5A, 0x11FC6, 0x11FC6, 0x11FC7, 0x11FC7, 0xD5B, 0xD5B, 0x11FC8, 0x11FC8, + 0x9F4, 0x9F4, 0xB75, 0xB75, 0xD76, 0xD76, 0xA833, 0xA833, 0x11FC9, 0x11FCA, + 0x109F6, 0x109F6, 0xD5C, 0xD5C, 0x2152, 0x2152, 0x11FCB, 0x11FCB, 0x2151, 0x2151, + 0x9F5, 0x9F5, 0xB76, 0xB76, 0xD77, 0xD77, 0x215B, 0x215B, 0xA834, 0xA834, + 0x11FCC, 0x11FCC, 0x1245F, 0x1245F, 0x2150, 0x2150, 0xD5D, 0xD5D, 0x11FCD, 0x11FCD, + 0x2159, 0x2159, 0x109F7, 0x109F7, 0x12461, 0x12461, 0x1ED3D, 0x1ED3D, 0x9F6, 0x9F6, + 0xB77, 0xB77, 0xD78, 0xD78, 0xA835, 0xA835, 0x11FCE, 0x11FCE, 0xD5E, 0xD5E, + 0x2155, 0x2155, 0x11FCF, 0x11FCF, 0xBC, 0xBC, 0x9F7, 0x9F7, 0xB72, 0xB72, + 0xD73, 0xD73, 0xA830, 0xA830, 0x10140, 0x10140, 0x1018B, 0x1018B, 0x109F8, 0x109F8, + 0x10E7C, 0x10E7C, 0x11FD0, 0x11FD0, 0x12460, 0x12460, 0x12462, 0x12463, 0x1ECAD, 0x1ECAD, + 0x2153, 0x2153, 0x109F9, 0x109F9, 0x10E7D, 0x10E7D, 0x1245A, 0x1245A, 0x1245D, 0x1245D, + 0x12465, 0x12465, 0x215C, 0x215C, 0x2156, 0x2156, 0x109FA, 0x109FA, 0xBD, 0xBD, + 0xB73, 0xB73, 0xD74, 0xD74, 0xF2A, 0xF2A, 0x2CFD, 0x2CFD, 0xA831, 0xA831, + 0x10141, 0x10141, 0x10175, 0x10176, 0x109BD, 0x109BD, 0x109FB, 0x109FB, 0x10A48, 0x10A48, + 0x10E7B, 0x10E7B, 0x10F26, 0x10F26, 0x11FD1, 0x11FD2, 0x12226, 0x12226, 0x12464, 0x12464, + 0x1ECAE, 0x1ECAE, 0x1ED3C, 0x1ED3C, 0x109FC, 0x109FC, 0x2157, 0x2157, 0x215D, 0x215D, + 0x2154, 0x2154, 0x10177, 0x10177, 0x109FD, 0x109FD, 0x10E7E, 0x10E7E, 0x1245B, 0x1245B, + 0x1245E, 0x1245E, 0x12466, 0x12466, 0xBE, 0xBE, 0x9F8, 0x9F8, 0xB74, 0xB74, + 0xD75, 0xD75, 0xA832, 0xA832, 0x10178, 0x10178, 0x109FE, 0x109FE, 0x11FD3, 0x11FD3, + 0x1ECAF, 0x1ECAF, 0x2158, 0x2158, 0x215A, 0x215A, 0x109FF, 0x109FF, 0x1245C, 0x1245C, + 0x215E, 0x215E, 0x109BC, 0x109BC, 0x31, 0x31, 0xB9, 0xB9, 0x661, 0x661, + 0x6F1, 0x6F1, 0x7C1, 0x7C1, 0x967, 0x967, 0x9E7, 0x9E7, 0xA67, 0xA67, + 0xAE7, 0xAE7, 0xB67, 0xB67, 0xBE7, 0xBE7, 0xC67, 0xC67, 0xC79, 0xC79, + 0xC7C, 0xC7C, 0xCE7, 0xCE7, 0xD67, 0xD67, 0xDE7, 0xDE7, 0xE51, 0xE51, + 0xED1, 0xED1, 0xF21, 0xF21, 0x1041, 0x1041, 0x1091, 0x1091, 0x1369, 0x1369, + 0x17E1, 0x17E1, 0x17F1, 0x17F1, 0x1811, 0x1811, 0x1947, 0x1947, 0x19D1, 0x19D1, + 0x19DA, 0x19DA, 0x1A81, 0x1A81, 0x1A91, 0x1A91, 0x1B51, 0x1B51, 0x1BB1, 0x1BB1, + 0x1C41, 0x1C41, 0x1C51, 0x1C51, 0x2081, 0x2081, 0x215F, 0x2160, 0x2170, 0x2170, + 0x2460, 0x2460, 0x2474, 0x2474, 0x2488, 0x2488, 0x24F5, 0x24F5, 0x2776, 0x2776, + 0x2780, 0x2780, 0x278A, 0x278A, 0x3021, 0x3021, 0x3192, 0x3192, 0x3220, 0x3220, + 0x3280, 0x3280, 0x4E00, 0x4E00, 0x58F1, 0x58F1, 0x58F9, 0x58F9, 0x5E7A, 0x5E7A, + 0x5F0C, 0x5F0C, 0xA621, 0xA621, 0xA6E6, 0xA6E6, 0xA8D1, 0xA8D1, 0xA901, 0xA901, + 0xA9D1, 0xA9D1, 0xA9F1, 0xA9F1, 0xAA51, 0xAA51, 0xABF1, 0xABF1, 0xFF11, 0xFF11, + 0x10107, 0x10107, 0x10142, 0x10142, 0x10158, 0x1015A, 0x102E1, 0x102E1, 0x10320, 0x10320, + 0x103D1, 0x103D1, 0x104A1, 0x104A1, 0x10858, 0x10858, 0x10879, 0x10879, 0x108A7, 0x108A7, + 0x108FB, 0x108FB, 0x10916, 0x10916, 0x109C0, 0x109C0, 0x10A40, 0x10A40, 0x10A7D, 0x10A7D, + 0x10A9D, 0x10A9D, 0x10AEB, 0x10AEB, 0x10B58, 0x10B58, 0x10B78, 0x10B78, 0x10BA9, 0x10BA9, + 0x10CFA, 0x10CFA, 0x10D31, 0x10D31, 0x10D41, 0x10D41, 0x10E60, 0x10E60, 0x10F1D, 0x10F1D, + 0x10F51, 0x10F51, 0x10FC5, 0x10FC5, 0x11052, 0x11052, 0x11067, 0x11067, 0x110F1, 0x110F1, + 0x11137, 0x11137, 0x111D1, 0x111D1, 0x111E1, 0x111E1, 0x112F1, 0x112F1, 0x11451, 0x11451, + 0x114D1, 0x114D1, 0x11651, 0x11651, 0x116C1, 0x116C1, 0x116D1, 0x116D1, 0x116DB, 0x116DB, + 0x11731, 0x11731, 0x118E1, 0x118E1, 0x11951, 0x11951, 0x11BF1, 0x11BF1, 0x11C51, 0x11C51, + 0x11C5A, 0x11C5A, 0x11D51, 0x11D51, 0x11DA1, 0x11DA1, 0x11DE1, 0x11DE1, 0x11F51, 0x11F51, + 0x12038, 0x12039, 0x12079, 0x12079, 0x1230B, 0x1230B, 0x12415, 0x12415, 0x1241E, 0x1241E, + 0x1242C, 0x1242C, 0x12434, 0x12434, 0x1244F, 0x1244F, 0x12458, 0x12458, 0x16131, 0x16131, + 0x16A61, 0x16A61, 0x16AC1, 0x16AC1, 0x16B51, 0x16B51, 0x16D71, 0x16D71, 0x16E81, 0x16E81, + 0x16E94, 0x16E94, 0x16FF4, 0x16FF4, 0x1CCF1, 0x1CCF1, 0x1D2C1, 0x1D2C1, 0x1D2E1, 0x1D2E1, + 0x1D360, 0x1D360, 0x1D372, 0x1D372, 0x1D377, 0x1D377, 0x1D7CF, 0x1D7CF, 0x1D7D9, 0x1D7D9, + 0x1D7E3, 0x1D7E3, 0x1D7ED, 0x1D7ED, 0x1D7F7, 0x1D7F7, 0x1E141, 0x1E141, 0x1E2F1, 0x1E2F1, + 0x1E4F1, 0x1E4F1, 0x1E5F2, 0x1E5F2, 0x1E8C7, 0x1E8C7, 0x1E951, 0x1E951, 0x1EC71, 0x1EC71, + 0x1ECA3, 0x1ECA3, 0x1ECB1, 0x1ECB1, 0x1ED01, 0x1ED01, 0x1F102, 0x1F102, 0x1FBF1, 0x1FBF1, + 0x2092A, 0x2092A, 0xF2B, 0xF2B, 0x16FF5, 0x16FF5, 0x32, 0x32, 0xB2, 0xB2, + 0x662, 0x662, 0x6F2, 0x6F2, 0x7C2, 0x7C2, 0x968, 0x968, 0x9E8, 0x9E8, + 0xA68, 0xA68, 0xAE8, 0xAE8, 0xB68, 0xB68, 0xBE8, 0xBE8, 0xC68, 0xC68, + 0xC7A, 0xC7A, 0xC7D, 0xC7D, 0xCE8, 0xCE8, 0xD68, 0xD68, 0xDE8, 0xDE8, + 0xE52, 0xE52, 0xED2, 0xED2, 0xF22, 0xF22, 0x1042, 0x1042, 0x1092, 0x1092, + 0x136A, 0x136A, 0x17E2, 0x17E2, 0x17F2, 0x17F2, 0x1812, 0x1812, 0x1948, 0x1948, + 0x19D2, 0x19D2, 0x1A82, 0x1A82, 0x1A92, 0x1A92, 0x1B52, 0x1B52, 0x1BB2, 0x1BB2, + 0x1C42, 0x1C42, 0x1C52, 0x1C52, 0x2082, 0x2082, 0x2161, 0x2161, 0x2171, 0x2171, + 0x2461, 0x2461, 0x2475, 0x2475, 0x2489, 0x2489, 0x24F6, 0x24F6, 0x2777, 0x2777, + 0x2781, 0x2781, 0x278B, 0x278B, 0x3022, 0x3022, 0x3193, 0x3193, 0x3221, 0x3221, + 0x3281, 0x3281, 0x3483, 0x3483, 0x4E24, 0x4E24, 0x4E8C, 0x4E8C, 0x4FE9, 0x4FE9, + 0x5006, 0x5006, 0x5169, 0x5169, 0x5F0D, 0x5F0D, 0x5F10, 0x5F10, 0x8CAE, 0x8CAE, + 0x8CB3, 0x8CB3, 0x8D30, 0x8D30, 0xA622, 0xA622, 0xA6E7, 0xA6E7, 0xA8D2, 0xA8D2, + 0xA902, 0xA902, 0xA9D2, 0xA9D2, 0xA9F2, 0xA9F2, 0xAA52, 0xAA52, 0xABF2, 0xABF2, + 0xF978, 0xF978, 0xFF12, 0xFF12, 0x10108, 0x10108, 0x1015B, 0x1015E, 0x102E2, 0x102E2, + 0x103D2, 0x103D2, 0x104A2, 0x104A2, 0x10859, 0x10859, 0x1087A, 0x1087A, 0x108A8, 0x108A8, + 0x1091A, 0x1091A, 0x109C1, 0x109C1, 0x10A41, 0x10A41, 0x10B59, 0x10B59, 0x10B79, 0x10B79, + 0x10BAA, 0x10BAA, 0x10D32, 0x10D32, 0x10D42, 0x10D42, 0x10E61, 0x10E61, 0x10F1E, 0x10F1E, + 0x10FC6, 0x10FC6, 0x11053, 0x11053, 0x11068, 0x11068, 0x110F2, 0x110F2, 0x11138, 0x11138, + 0x111D2, 0x111D2, 0x111E2, 0x111E2, 0x112F2, 0x112F2, 0x11452, 0x11452, 0x114D2, 0x114D2, + 0x11652, 0x11652, 0x116C2, 0x116C2, 0x116D2, 0x116D2, 0x116DC, 0x116DC, 0x11732, 0x11732, + 0x118E2, 0x118E2, 0x11952, 0x11952, 0x11BF2, 0x11BF2, 0x11C52, 0x11C52, 0x11C5B, 0x11C5B, + 0x11D52, 0x11D52, 0x11DA2, 0x11DA2, 0x11DE2, 0x11DE2, 0x11F52, 0x11F52, 0x1222B, 0x1222B, + 0x12399, 0x12399, 0x12400, 0x12400, 0x12416, 0x12416, 0x1241F, 0x1241F, 0x12423, 0x12423, + 0x1242D, 0x1242D, 0x12435, 0x12435, 0x1244A, 0x1244A, 0x12450, 0x12450, 0x12456, 0x12456, + 0x12459, 0x12459, 0x16132, 0x16132, 0x16A62, 0x16A62, 0x16AC2, 0x16AC2, 0x16B52, 0x16B52, + 0x16D72, 0x16D72, 0x16E82, 0x16E82, 0x16E95, 0x16E95, 0x16FF6, 0x16FF6, 0x1CCF2, 0x1CCF2, + 0x1D2C2, 0x1D2C2, 0x1D2E2, 0x1D2E2, 0x1D361, 0x1D361, 0x1D373, 0x1D373, 0x1D7D0, 0x1D7D0, + 0x1D7DA, 0x1D7DA, 0x1D7E4, 0x1D7E4, 0x1D7EE, 0x1D7EE, 0x1D7F8, 0x1D7F8, 0x1E142, 0x1E142, + 0x1E2F2, 0x1E2F2, 0x1E4F2, 0x1E4F2, 0x1E5F3, 0x1E5F3, 0x1E8C8, 0x1E8C8, 0x1E952, 0x1E952, + 0x1EC72, 0x1EC72, 0x1ECA4, 0x1ECA4, 0x1ECB2, 0x1ECB2, 0x1ED02, 0x1ED02, 0x1ED2F, 0x1ED2F, + 0x1F103, 0x1F103, 0x1FBF2, 0x1FBF2, 0x22390, 0x22390, 0xF2C, 0xF2C, 0x33, 0x33, + 0xB3, 0xB3, 0x663, 0x663, 0x6F3, 0x6F3, 0x7C3, 0x7C3, 0x969, 0x969, + 0x9E9, 0x9E9, 0xA69, 0xA69, 0xAE9, 0xAE9, 0xB69, 0xB69, 0xBE9, 0xBE9, + 0xC69, 0xC69, 0xC7B, 0xC7B, 0xC7E, 0xC7E, 0xCE9, 0xCE9, 0xD69, 0xD69, + 0xDE9, 0xDE9, 0xE53, 0xE53, 0xED3, 0xED3, 0xF23, 0xF23, 0x1043, 0x1043, + 0x1093, 0x1093, 0x136B, 0x136B, 0x17E3, 0x17E3, 0x17F3, 0x17F3, 0x1813, 0x1813, + 0x1949, 0x1949, 0x19D3, 0x19D3, 0x1A83, 0x1A83, 0x1A93, 0x1A93, 0x1B53, 0x1B53, + 0x1BB3, 0x1BB3, 0x1C43, 0x1C43, 0x1C53, 0x1C53, 0x2083, 0x2083, 0x2162, 0x2162, + 0x2172, 0x2172, 0x2462, 0x2462, 0x2476, 0x2476, 0x248A, 0x248A, 0x24F7, 0x24F7, + 0x2778, 0x2778, 0x2782, 0x2782, 0x278C, 0x278C, 0x3023, 0x3023, 0x3194, 0x3194, + 0x3222, 0x3222, 0x3282, 0x3282, 0x4E09, 0x4E09, 0x4EE8, 0x4EE8, 0x53C1, 0x53C4, + 0x5F0E, 0x5F0E, 0xA623, 0xA623, 0xA6E8, 0xA6E8, 0xA8D3, 0xA8D3, 0xA903, 0xA903, + 0xA9D3, 0xA9D3, 0xA9F3, 0xA9F3, 0xAA53, 0xAA53, 0xABF3, 0xABF3, 0xF96B, 0xF96B, + 0xFF13, 0xFF13, 0x10109, 0x10109, 0x102E3, 0x102E3, 0x104A3, 0x104A3, 0x1085A, 0x1085A, + 0x1087B, 0x1087B, 0x108A9, 0x108A9, 0x1091B, 0x1091B, 0x109C2, 0x109C2, 0x10A42, 0x10A42, + 0x10B5A, 0x10B5A, 0x10B7A, 0x10B7A, 0x10BAB, 0x10BAB, 0x10D33, 0x10D33, 0x10D43, 0x10D43, + 0x10E62, 0x10E62, 0x10F1F, 0x10F1F, 0x10FC7, 0x10FC7, 0x11054, 0x11054, 0x11069, 0x11069, + 0x110F3, 0x110F3, 0x11139, 0x11139, 0x111D3, 0x111D3, 0x111E3, 0x111E3, 0x112F3, 0x112F3, + 0x11453, 0x11453, 0x114D3, 0x114D3, 0x11653, 0x11653, 0x116C3, 0x116C3, 0x116D3, 0x116D3, + 0x116DD, 0x116DD, 0x11733, 0x11733, 0x118E3, 0x118E3, 0x11953, 0x11953, 0x11BF3, 0x11BF3, + 0x11C53, 0x11C53, 0x11C5C, 0x11C5C, 0x11D53, 0x11D53, 0x11DA3, 0x11DA3, 0x11DE3, 0x11DE3, + 0x11F53, 0x11F53, 0x1230D, 0x1230D, 0x12401, 0x12401, 0x12408, 0x12408, 0x12417, 0x12417, + 0x12420, 0x12420, 0x12424, 0x12425, 0x1242E, 0x1242F, 0x12436, 0x12437, 0x1243A, 0x1243B, + 0x1244B, 0x1244B, 0x12451, 0x12451, 0x12457, 0x12457, 0x16133, 0x16133, 0x16A63, 0x16A63, + 0x16AC3, 0x16AC3, 0x16B53, 0x16B53, 0x16D73, 0x16D73, 0x16E83, 0x16E83, 0x16E96, 0x16E96, + 0x1CCF3, 0x1CCF3, 0x1D2C3, 0x1D2C3, 0x1D2E3, 0x1D2E3, 0x1D362, 0x1D362, 0x1D374, 0x1D374, + 0x1D7D1, 0x1D7D1, 0x1D7DB, 0x1D7DB, 0x1D7E5, 0x1D7E5, 0x1D7EF, 0x1D7EF, 0x1D7F9, 0x1D7F9, + 0x1E143, 0x1E143, 0x1E2F3, 0x1E2F3, 0x1E4F3, 0x1E4F3, 0x1E5F4, 0x1E5F4, 0x1E8C9, 0x1E8C9, + 0x1E953, 0x1E953, 0x1EC73, 0x1EC73, 0x1ECA5, 0x1ECA5, 0x1ED03, 0x1ED03, 0x1ED30, 0x1ED30, + 0x1F104, 0x1F104, 0x1FBF3, 0x1FBF3, 0x20AFD, 0x20AFD, 0x20B19, 0x20B19, 0x22998, 0x22998, + 0x23B1B, 0x23B1B, 0xF2D, 0xF2D, 0x34, 0x34, 0x664, 0x664, 0x6F4, 0x6F4, + 0x7C4, 0x7C4, 0x96A, 0x96A, 0x9EA, 0x9EA, 0xA6A, 0xA6A, 0xAEA, 0xAEA, + 0xB6A, 0xB6A, 0xBEA, 0xBEA, 0xC6A, 0xC6A, 0xCEA, 0xCEA, 0xD6A, 0xD6A, + 0xDEA, 0xDEA, 0xE54, 0xE54, 0xED4, 0xED4, 0xF24, 0xF24, 0x1044, 0x1044, + 0x1094, 0x1094, 0x136C, 0x136C, 0x17E4, 0x17E4, 0x17F4, 0x17F4, 0x1814, 0x1814, + 0x194A, 0x194A, 0x19D4, 0x19D4, 0x1A84, 0x1A84, 0x1A94, 0x1A94, 0x1B54, 0x1B54, + 0x1BB4, 0x1BB4, 0x1C44, 0x1C44, 0x1C54, 0x1C54, 0x2074, 0x2074, 0x2084, 0x2084, + 0x2163, 0x2163, 0x2173, 0x2173, 0x2463, 0x2463, 0x2477, 0x2477, 0x248B, 0x248B, + 0x24F8, 0x24F8, 0x2779, 0x2779, 0x2783, 0x2783, 0x278D, 0x278D, 0x3024, 0x3024, + 0x3195, 0x3195, 0x3223, 0x3223, 0x3283, 0x3283, 0x4E96, 0x4E96, 0x56DB, 0x56DB, + 0x8086, 0x8086, 0xA624, 0xA624, 0xA6E9, 0xA6E9, 0xA8D4, 0xA8D4, 0xA904, 0xA904, + 0xA9D4, 0xA9D4, 0xA9F4, 0xA9F4, 0xAA54, 0xAA54, 0xABF4, 0xABF4, 0xFF14, 0xFF14, + 0x1010A, 0x1010A, 0x102E4, 0x102E4, 0x104A4, 0x104A4, 0x1087C, 0x1087C, 0x108AA, 0x108AB, + 0x109C3, 0x109C3, 0x10A43, 0x10A43, 0x10B5B, 0x10B5B, 0x10B7B, 0x10B7B, 0x10BAC, 0x10BAC, + 0x10D34, 0x10D34, 0x10D44, 0x10D44, 0x10E63, 0x10E63, 0x10F20, 0x10F20, 0x10FC8, 0x10FC8, + 0x11055, 0x11055, 0x1106A, 0x1106A, 0x110F4, 0x110F4, 0x1113A, 0x1113A, 0x111D4, 0x111D4, + 0x111E4, 0x111E4, 0x112F4, 0x112F4, 0x11454, 0x11454, 0x114D4, 0x114D4, 0x11654, 0x11654, + 0x116C4, 0x116C4, 0x116D4, 0x116D4, 0x116DE, 0x116DE, 0x11734, 0x11734, 0x118E4, 0x118E4, + 0x11954, 0x11954, 0x11BF4, 0x11BF4, 0x11C54, 0x11C54, 0x11C5D, 0x11C5D, 0x11D54, 0x11D54, + 0x11DA4, 0x11DA4, 0x11DE4, 0x11DE4, 0x11F54, 0x11F54, 0x12402, 0x12402, 0x12409, 0x12409, + 0x1240F, 0x1240F, 0x12418, 0x12418, 0x12421, 0x12421, 0x12426, 0x12426, 0x12430, 0x12430, + 0x12438, 0x12438, 0x1243C, 0x1243F, 0x1244C, 0x1244C, 0x12452, 0x12453, 0x12469, 0x12469, + 0x16134, 0x16134, 0x16A64, 0x16A64, 0x16AC4, 0x16AC4, 0x16B54, 0x16B54, 0x16D74, 0x16D74, + 0x16E84, 0x16E84, 0x1CCF4, 0x1CCF4, 0x1D2C4, 0x1D2C4, 0x1D2E4, 0x1D2E4, 0x1D363, 0x1D363, + 0x1D375, 0x1D375, 0x1D7D2, 0x1D7D2, 0x1D7DC, 0x1D7DC, 0x1D7E6, 0x1D7E6, 0x1D7F0, 0x1D7F0, + 0x1D7FA, 0x1D7FA, 0x1E144, 0x1E144, 0x1E2F4, 0x1E2F4, 0x1E4F4, 0x1E4F4, 0x1E5F5, 0x1E5F5, + 0x1E8CA, 0x1E8CA, 0x1E954, 0x1E954, 0x1EC74, 0x1EC74, 0x1ECA6, 0x1ECA6, 0x1ED04, 0x1ED04, + 0x1ED31, 0x1ED31, 0x1F105, 0x1F105, 0x1FBF4, 0x1FBF4, 0x20064, 0x20064, 0x200E2, 0x200E2, + 0x2626D, 0x2626D, 0xF2E, 0xF2E, 0x35, 0x35, 0x665, 0x665, 0x6F5, 0x6F5, + 0x7C5, 0x7C5, 0x96B, 0x96B, 0x9EB, 0x9EB, 0xA6B, 0xA6B, 0xAEB, 0xAEB, + 0xB6B, 0xB6B, 0xBEB, 0xBEB, 0xC6B, 0xC6B, 0xCEB, 0xCEB, 0xD6B, 0xD6B, + 0xDEB, 0xDEB, 0xE55, 0xE55, 0xED5, 0xED5, 0xF25, 0xF25, 0x1045, 0x1045, + 0x1095, 0x1095, 0x136D, 0x136D, 0x17E5, 0x17E5, 0x17F5, 0x17F5, 0x1815, 0x1815, + 0x194B, 0x194B, 0x19D5, 0x19D5, 0x1A85, 0x1A85, 0x1A95, 0x1A95, 0x1B55, 0x1B55, + 0x1BB5, 0x1BB5, 0x1C45, 0x1C45, 0x1C55, 0x1C55, 0x2075, 0x2075, 0x2085, 0x2085, + 0x2164, 0x2164, 0x2174, 0x2174, 0x2464, 0x2464, 0x2478, 0x2478, 0x248C, 0x248C, + 0x24F9, 0x24F9, 0x277A, 0x277A, 0x2784, 0x2784, 0x278E, 0x278E, 0x3025, 0x3025, + 0x3224, 0x3224, 0x3284, 0x3284, 0x3405, 0x3405, 0x382A, 0x382A, 0x4E94, 0x4E94, + 0x4F0D, 0x4F0D, 0xA625, 0xA625, 0xA6EA, 0xA6EA, 0xA8D5, 0xA8D5, 0xA905, 0xA905, + 0xA9D5, 0xA9D5, 0xA9F5, 0xA9F5, 0xAA55, 0xAA55, 0xABF5, 0xABF5, 0xFF15, 0xFF15, + 0x1010B, 0x1010B, 0x10143, 0x10143, 0x10148, 0x10148, 0x1014F, 0x1014F, 0x1015F, 0x1015F, + 0x10173, 0x10173, 0x102E5, 0x102E5, 0x10321, 0x10321, 0x104A5, 0x104A5, 0x1087D, 0x1087D, + 0x108AC, 0x108AC, 0x108FC, 0x108FC, 0x109C4, 0x109C4, 0x10AEC, 0x10AEC, 0x10CFB, 0x10CFB, + 0x10D35, 0x10D35, 0x10D45, 0x10D45, 0x10E64, 0x10E64, 0x10F21, 0x10F21, 0x11056, 0x11056, + 0x1106B, 0x1106B, 0x110F5, 0x110F5, 0x1113B, 0x1113B, 0x111D5, 0x111D5, 0x111E5, 0x111E5, + 0x112F5, 0x112F5, 0x11455, 0x11455, 0x114D5, 0x114D5, 0x11655, 0x11655, 0x116C5, 0x116C5, + 0x116D5, 0x116D5, 0x116DF, 0x116DF, 0x11735, 0x11735, 0x118E5, 0x118E5, 0x11955, 0x11955, + 0x11BF5, 0x11BF5, 0x11C55, 0x11C55, 0x11C5E, 0x11C5E, 0x11D55, 0x11D55, 0x11DA5, 0x11DA5, + 0x11DE5, 0x11DE5, 0x11F55, 0x11F55, 0x12403, 0x12403, 0x1240A, 0x1240A, 0x12410, 0x12410, + 0x12419, 0x12419, 0x12422, 0x12422, 0x12427, 0x12427, 0x12431, 0x12431, 0x12439, 0x12439, + 0x1244D, 0x1244D, 0x12454, 0x12455, 0x1246A, 0x1246A, 0x16135, 0x16135, 0x16A65, 0x16A65, + 0x16AC5, 0x16AC5, 0x16B55, 0x16B55, 0x16D75, 0x16D75, 0x16E85, 0x16E85, 0x1CCF5, 0x1CCF5, + 0x1D2C5, 0x1D2C5, 0x1D2E5, 0x1D2E5, 0x1D364, 0x1D364, 0x1D376, 0x1D376, 0x1D378, 0x1D378, + 0x1D7D3, 0x1D7D3, 0x1D7DD, 0x1D7DD, 0x1D7E7, 0x1D7E7, 0x1D7F1, 0x1D7F1, 0x1D7FB, 0x1D7FB, + 0x1E145, 0x1E145, 0x1E2F5, 0x1E2F5, 0x1E4F5, 0x1E4F5, 0x1E5F6, 0x1E5F6, 0x1E8CB, 0x1E8CB, + 0x1E955, 0x1E955, 0x1EC75, 0x1EC75, 0x1ECA7, 0x1ECA7, 0x1ED05, 0x1ED05, 0x1ED32, 0x1ED32, + 0x1F106, 0x1F106, 0x1FBF5, 0x1FBF5, 0x20121, 0x20121, 0xF2F, 0xF2F, 0x36, 0x36, + 0x666, 0x666, 0x6F6, 0x6F6, 0x7C6, 0x7C6, 0x96C, 0x96C, 0x9EC, 0x9EC, + 0xA6C, 0xA6C, 0xAEC, 0xAEC, 0xB6C, 0xB6C, 0xBEC, 0xBEC, 0xC6C, 0xC6C, + 0xCEC, 0xCEC, 0xD6C, 0xD6C, 0xDEC, 0xDEC, 0xE56, 0xE56, 0xED6, 0xED6, + 0xF26, 0xF26, 0x1046, 0x1046, 0x1096, 0x1096, 0x136E, 0x136E, 0x17E6, 0x17E6, + 0x17F6, 0x17F6, 0x1816, 0x1816, 0x194C, 0x194C, 0x19D6, 0x19D6, 0x1A86, 0x1A86, + 0x1A96, 0x1A96, 0x1B56, 0x1B56, 0x1BB6, 0x1BB6, 0x1C46, 0x1C46, 0x1C56, 0x1C56, + 0x2076, 0x2076, 0x2086, 0x2086, 0x2165, 0x2165, 0x2175, 0x2175, 0x2185, 0x2185, + 0x2465, 0x2465, 0x2479, 0x2479, 0x248D, 0x248D, 0x24FA, 0x24FA, 0x277B, 0x277B, + 0x2785, 0x2785, 0x278F, 0x278F, 0x3026, 0x3026, 0x3225, 0x3225, 0x3285, 0x3285, + 0x516D, 0x516D, 0x9646, 0x9646, 0x9678, 0x9678, 0xA626, 0xA626, 0xA6EB, 0xA6EB, + 0xA8D6, 0xA8D6, 0xA906, 0xA906, 0xA9D6, 0xA9D6, 0xA9F6, 0xA9F6, 0xAA56, 0xAA56, + 0xABF6, 0xABF6, 0xF9D1, 0xF9D1, 0xF9D3, 0xF9D3, 0xFF16, 0xFF16, 0x1010C, 0x1010C, + 0x102E6, 0x102E6, 0x104A6, 0x104A6, 0x109C5, 0x109C5, 0x10D36, 0x10D36, 0x10D46, 0x10D46, + 0x10E65, 0x10E65, 0x11057, 0x11057, 0x1106C, 0x1106C, 0x110F6, 0x110F6, 0x1113C, 0x1113C, + 0x111D6, 0x111D6, 0x111E6, 0x111E6, 0x112F6, 0x112F6, 0x11456, 0x11456, 0x114D6, 0x114D6, + 0x11656, 0x11656, 0x116C6, 0x116C6, 0x116D6, 0x116D6, 0x116E0, 0x116E0, 0x11736, 0x11736, + 0x118E6, 0x118E6, 0x11956, 0x11956, 0x11BF6, 0x11BF6, 0x11C56, 0x11C56, 0x11C5F, 0x11C5F, + 0x11D56, 0x11D56, 0x11DA6, 0x11DA6, 0x11DE6, 0x11DE6, 0x11F56, 0x11F56, 0x12404, 0x12404, + 0x1240B, 0x1240B, 0x12411, 0x12411, 0x1241A, 0x1241A, 0x12428, 0x12428, 0x12440, 0x12440, + 0x1244E, 0x1244E, 0x1246B, 0x1246B, 0x16136, 0x16136, 0x16A66, 0x16A66, 0x16AC6, 0x16AC6, + 0x16B56, 0x16B56, 0x16D76, 0x16D76, 0x16E86, 0x16E86, 0x1CCF6, 0x1CCF6, 0x1D2C6, 0x1D2C6, + 0x1D2E6, 0x1D2E6, 0x1D365, 0x1D365, 0x1D7D4, 0x1D7D4, 0x1D7DE, 0x1D7DE, 0x1D7E8, 0x1D7E8, + 0x1D7F2, 0x1D7F2, 0x1D7FC, 0x1D7FC, 0x1E146, 0x1E146, 0x1E2F6, 0x1E2F6, 0x1E4F6, 0x1E4F6, + 0x1E5F7, 0x1E5F7, 0x1E8CC, 0x1E8CC, 0x1E956, 0x1E956, 0x1EC76, 0x1EC76, 0x1ECA8, 0x1ECA8, + 0x1ED06, 0x1ED06, 0x1ED33, 0x1ED33, 0x1F107, 0x1F107, 0x1FBF6, 0x1FBF6, 0x20AEA, 0x20AEA, + 0xF30, 0xF30, 0x37, 0x37, 0x667, 0x667, 0x6F7, 0x6F7, 0x7C7, 0x7C7, + 0x96D, 0x96D, 0x9ED, 0x9ED, 0xA6D, 0xA6D, 0xAED, 0xAED, 0xB6D, 0xB6D, + 0xBED, 0xBED, 0xC6D, 0xC6D, 0xCED, 0xCED, 0xD6D, 0xD6D, 0xDED, 0xDED, + 0xE57, 0xE57, 0xED7, 0xED7, 0xF27, 0xF27, 0x1047, 0x1047, 0x1097, 0x1097, + 0x136F, 0x136F, 0x17E7, 0x17E7, 0x17F7, 0x17F7, 0x1817, 0x1817, 0x194D, 0x194D, + 0x19D7, 0x19D7, 0x1A87, 0x1A87, 0x1A97, 0x1A97, 0x1B57, 0x1B57, 0x1BB7, 0x1BB7, + 0x1C47, 0x1C47, 0x1C57, 0x1C57, 0x2077, 0x2077, 0x2087, 0x2087, 0x2166, 0x2166, + 0x2176, 0x2176, 0x2466, 0x2466, 0x247A, 0x247A, 0x248E, 0x248E, 0x24FB, 0x24FB, + 0x277C, 0x277C, 0x2786, 0x2786, 0x2790, 0x2790, 0x3027, 0x3027, 0x3226, 0x3226, + 0x3286, 0x3286, 0x3B4D, 0x3B4D, 0x4E03, 0x4E03, 0x62D0, 0x62D0, 0x67D2, 0x67D2, + 0x6F06, 0x6F06, 0xA627, 0xA627, 0xA6EC, 0xA6EC, 0xA8D7, 0xA8D7, 0xA907, 0xA907, + 0xA9D7, 0xA9D7, 0xA9F7, 0xA9F7, 0xAA57, 0xAA57, 0xABF7, 0xABF7, 0xFF17, 0xFF17, + 0x1010D, 0x1010D, 0x102E7, 0x102E7, 0x104A7, 0x104A7, 0x109C6, 0x109C6, 0x10D37, 0x10D37, + 0x10D47, 0x10D47, 0x10E66, 0x10E66, 0x11058, 0x11058, 0x1106D, 0x1106D, 0x110F7, 0x110F7, + 0x1113D, 0x1113D, 0x111D7, 0x111D7, 0x111E7, 0x111E7, 0x112F7, 0x112F7, 0x11457, 0x11457, + 0x114D7, 0x114D7, 0x11657, 0x11657, 0x116C7, 0x116C7, 0x116D7, 0x116D7, 0x116E1, 0x116E1, + 0x11737, 0x11737, 0x118E7, 0x118E7, 0x11957, 0x11957, 0x11BF7, 0x11BF7, 0x11C57, 0x11C57, + 0x11C60, 0x11C60, 0x11D57, 0x11D57, 0x11DA7, 0x11DA7, 0x11DE7, 0x11DE7, 0x11F57, 0x11F57, + 0x12405, 0x12405, 0x1240C, 0x1240C, 0x12412, 0x12412, 0x1241B, 0x1241B, 0x12429, 0x12429, + 0x12441, 0x12443, 0x1246C, 0x1246C, 0x16137, 0x16137, 0x16A67, 0x16A67, 0x16AC7, 0x16AC7, + 0x16B57, 0x16B57, 0x16D77, 0x16D77, 0x16E87, 0x16E87, 0x1CCF7, 0x1CCF7, 0x1D2C7, 0x1D2C7, + 0x1D2E7, 0x1D2E7, 0x1D366, 0x1D366, 0x1D7D5, 0x1D7D5, 0x1D7DF, 0x1D7DF, 0x1D7E9, 0x1D7E9, + 0x1D7F3, 0x1D7F3, 0x1D7FD, 0x1D7FD, 0x1E147, 0x1E147, 0x1E2F7, 0x1E2F7, 0x1E4F7, 0x1E4F7, + 0x1E5F8, 0x1E5F8, 0x1E8CD, 0x1E8CD, 0x1E957, 0x1E957, 0x1EC77, 0x1EC77, 0x1ECA9, 0x1ECA9, + 0x1ED07, 0x1ED07, 0x1ED34, 0x1ED34, 0x1F108, 0x1F108, 0x1FBF7, 0x1FBF7, 0x20001, 0x20001, + 0xF31, 0xF31, 0x38, 0x38, 0x668, 0x668, 0x6F8, 0x6F8, 0x7C8, 0x7C8, + 0x96E, 0x96E, 0x9EE, 0x9EE, 0xA6E, 0xA6E, 0xAEE, 0xAEE, 0xB6E, 0xB6E, + 0xBEE, 0xBEE, 0xC6E, 0xC6E, 0xCEE, 0xCEE, 0xD6E, 0xD6E, 0xDEE, 0xDEE, + 0xE58, 0xE58, 0xED8, 0xED8, 0xF28, 0xF28, 0x1048, 0x1048, 0x1098, 0x1098, + 0x1370, 0x1370, 0x17E8, 0x17E8, 0x17F8, 0x17F8, 0x1818, 0x1818, 0x194E, 0x194E, + 0x19D8, 0x19D8, 0x1A88, 0x1A88, 0x1A98, 0x1A98, 0x1B58, 0x1B58, 0x1BB8, 0x1BB8, + 0x1C48, 0x1C48, 0x1C58, 0x1C58, 0x2078, 0x2078, 0x2088, 0x2088, 0x2167, 0x2167, + 0x2177, 0x2177, 0x2467, 0x2467, 0x247B, 0x247B, 0x248F, 0x248F, 0x24FC, 0x24FC, + 0x277D, 0x277D, 0x2787, 0x2787, 0x2791, 0x2791, 0x3028, 0x3028, 0x3227, 0x3227, + 0x3287, 0x3287, 0x516B, 0x516B, 0x634C, 0x634C, 0xA628, 0xA628, 0xA6ED, 0xA6ED, + 0xA8D8, 0xA8D8, 0xA908, 0xA908, 0xA9D8, 0xA9D8, 0xA9F8, 0xA9F8, 0xAA58, 0xAA58, + 0xABF8, 0xABF8, 0xFF18, 0xFF18, 0x1010E, 0x1010E, 0x102E8, 0x102E8, 0x104A8, 0x104A8, + 0x109C7, 0x109C7, 0x10D38, 0x10D38, 0x10D48, 0x10D48, 0x10E67, 0x10E67, 0x11059, 0x11059, + 0x1106E, 0x1106E, 0x110F8, 0x110F8, 0x1113E, 0x1113E, 0x111D8, 0x111D8, 0x111E8, 0x111E8, + 0x112F8, 0x112F8, 0x11458, 0x11458, 0x114D8, 0x114D8, 0x11658, 0x11658, 0x116C8, 0x116C8, + 0x116D8, 0x116D8, 0x116E2, 0x116E2, 0x11738, 0x11738, 0x118E8, 0x118E8, 0x11958, 0x11958, + 0x11BF8, 0x11BF8, 0x11C58, 0x11C58, 0x11C61, 0x11C61, 0x11D58, 0x11D58, 0x11DA8, 0x11DA8, + 0x11DE8, 0x11DE8, 0x11F58, 0x11F58, 0x12406, 0x12406, 0x1240D, 0x1240D, 0x12413, 0x12413, + 0x1241C, 0x1241C, 0x1242A, 0x1242A, 0x12444, 0x12445, 0x1246D, 0x1246D, 0x16138, 0x16138, + 0x16A68, 0x16A68, 0x16AC8, 0x16AC8, 0x16B58, 0x16B58, 0x16D78, 0x16D78, 0x16E88, 0x16E88, + 0x1CCF8, 0x1CCF8, 0x1D2C8, 0x1D2C8, 0x1D2E8, 0x1D2E8, 0x1D367, 0x1D367, 0x1D7D6, 0x1D7D6, + 0x1D7E0, 0x1D7E0, 0x1D7EA, 0x1D7EA, 0x1D7F4, 0x1D7F4, 0x1D7FE, 0x1D7FE, 0x1E148, 0x1E148, + 0x1E2F8, 0x1E2F8, 0x1E4F8, 0x1E4F8, 0x1E5F9, 0x1E5F9, 0x1E8CE, 0x1E8CE, 0x1E958, 0x1E958, + 0x1EC78, 0x1EC78, 0x1ECAA, 0x1ECAA, 0x1ED08, 0x1ED08, 0x1ED35, 0x1ED35, 0x1F109, 0x1F109, + 0x1FBF8, 0x1FBF8, 0xF32, 0xF32, 0x39, 0x39, 0x669, 0x669, 0x6F9, 0x6F9, + 0x7C9, 0x7C9, 0x96F, 0x96F, 0x9EF, 0x9EF, 0xA6F, 0xA6F, 0xAEF, 0xAEF, + 0xB6F, 0xB6F, 0xBEF, 0xBEF, 0xC6F, 0xC6F, 0xCEF, 0xCEF, 0xD6F, 0xD6F, + 0xDEF, 0xDEF, 0xE59, 0xE59, 0xED9, 0xED9, 0xF29, 0xF29, 0x1049, 0x1049, + 0x1099, 0x1099, 0x1371, 0x1371, 0x17E9, 0x17E9, 0x17F9, 0x17F9, 0x1819, 0x1819, + 0x194F, 0x194F, 0x19D9, 0x19D9, 0x1A89, 0x1A89, 0x1A99, 0x1A99, 0x1B59, 0x1B59, + 0x1BB9, 0x1BB9, 0x1C49, 0x1C49, 0x1C59, 0x1C59, 0x2079, 0x2079, 0x2089, 0x2089, + 0x2168, 0x2168, 0x2178, 0x2178, 0x2468, 0x2468, 0x247C, 0x247C, 0x2490, 0x2490, + 0x24FD, 0x24FD, 0x277E, 0x277E, 0x2788, 0x2788, 0x2792, 0x2792, 0x3029, 0x3029, + 0x3228, 0x3228, 0x3288, 0x3288, 0x4E5D, 0x4E5D, 0x5EFE, 0x5EFE, 0x7396, 0x7396, + 0x920E, 0x920E, 0x94A9, 0x94A9, 0xA629, 0xA629, 0xA6EE, 0xA6EE, 0xA8D9, 0xA8D9, + 0xA909, 0xA909, 0xA9D9, 0xA9D9, 0xA9F9, 0xA9F9, 0xAA59, 0xAA59, 0xABF9, 0xABF9, + 0xFF19, 0xFF19, 0x1010F, 0x1010F, 0x102E9, 0x102E9, 0x104A9, 0x104A9, 0x109C8, 0x109C8, + 0x10D39, 0x10D39, 0x10D49, 0x10D49, 0x10E68, 0x10E68, 0x1105A, 0x1105A, 0x1106F, 0x1106F, + 0x110F9, 0x110F9, 0x1113F, 0x1113F, 0x111D9, 0x111D9, 0x111E9, 0x111E9, 0x112F9, 0x112F9, + 0x11459, 0x11459, 0x114D9, 0x114D9, 0x11659, 0x11659, 0x116C9, 0x116C9, 0x116D9, 0x116D9, + 0x116E3, 0x116E3, 0x11739, 0x11739, 0x118E9, 0x118E9, 0x11959, 0x11959, 0x11BF9, 0x11BF9, + 0x11C59, 0x11C59, 0x11C62, 0x11C62, 0x11D59, 0x11D59, 0x11DA9, 0x11DA9, 0x11DE9, 0x11DE9, + 0x11F59, 0x11F59, 0x12407, 0x12407, 0x1240E, 0x1240E, 0x12414, 0x12414, 0x1241D, 0x1241D, + 0x1242B, 0x1242B, 0x12446, 0x12449, 0x1246E, 0x1246E, 0x16139, 0x16139, 0x16A69, 0x16A69, + 0x16AC9, 0x16AC9, 0x16B59, 0x16B59, 0x16D79, 0x16D79, 0x16E89, 0x16E89, 0x1CCF9, 0x1CCF9, + 0x1D2C9, 0x1D2C9, 0x1D2E9, 0x1D2E9, 0x1D368, 0x1D368, 0x1D7D7, 0x1D7D7, 0x1D7E1, 0x1D7E1, + 0x1D7EB, 0x1D7EB, 0x1D7F5, 0x1D7F5, 0x1D7FF, 0x1D7FF, 0x1E149, 0x1E149, 0x1E2F9, 0x1E2F9, + 0x1E4F9, 0x1E4F9, 0x1E5FA, 0x1E5FA, 0x1E8CF, 0x1E8CF, 0x1E959, 0x1E959, 0x1EC79, 0x1EC79, + 0x1ECAB, 0x1ECAB, 0x1ED09, 0x1ED09, 0x1ED36, 0x1ED36, 0x1F10A, 0x1F10A, 0x1FBF9, 0x1FBF9, + 0x2F890, 0x2F890, 0xBF0, 0xBF0, 0xD70, 0xD70, 0x1372, 0x1372, 0x2169, 0x2169, + 0x2179, 0x2179, 0x2469, 0x2469, 0x247D, 0x247D, 0x2491, 0x2491, 0x24FE, 0x24FE, + 0x277F, 0x277F, 0x2789, 0x2789, 0x2793, 0x2793, 0x3038, 0x3038, 0x3229, 0x3229, + 0x3248, 0x3248, 0x3289, 0x3289, 0x4EC0, 0x4EC0, 0x5341, 0x5341, 0x62FE, 0x62FE, + 0xF973, 0xF973, 0xF9FD, 0xF9FD, 0x10110, 0x10110, 0x10149, 0x10149, 0x10150, 0x10150, + 0x10157, 0x10157, 0x10160, 0x10164, 0x102EA, 0x102EA, 0x10322, 0x10322, 0x103D3, 0x103D3, + 0x1085B, 0x1085B, 0x1087E, 0x1087E, 0x108AD, 0x108AD, 0x108FD, 0x108FD, 0x10917, 0x10917, + 0x109C9, 0x109C9, 0x10A44, 0x10A44, 0x10A9E, 0x10A9E, 0x10AED, 0x10AED, 0x10B5C, 0x10B5C, + 0x10B7C, 0x10B7C, 0x10BAD, 0x10BAD, 0x10CFC, 0x10CFC, 0x10E69, 0x10E69, 0x10F22, 0x10F22, + 0x10F52, 0x10F52, 0x10FC9, 0x10FC9, 0x1105B, 0x1105B, 0x111EA, 0x111EA, 0x1173A, 0x1173A, + 0x118EA, 0x118EA, 0x11C63, 0x11C63, 0x16B5B, 0x16B5B, 0x16E8A, 0x16E8A, 0x1D2CA, 0x1D2CA, + 0x1D2EA, 0x1D2EA, 0x1D369, 0x1D369, 0x1EC7A, 0x1EC7A, 0x1ED0A, 0x1ED0A, 0x1ED37, 0x1ED37, + 0x216A, 0x216A, 0x217A, 0x217A, 0x246A, 0x246A, 0x247E, 0x247E, 0x2492, 0x2492, + 0x24EB, 0x24EB, 0x16E8B, 0x16E8B, 0x1D2CB, 0x1D2CB, 0x1D2EB, 0x1D2EB, 0x216B, 0x216B, + 0x217B, 0x217B, 0x246B, 0x246B, 0x247F, 0x247F, 0x2493, 0x2493, 0x24EC, 0x24EC, + 0x16E8C, 0x16E8C, 0x1D2CC, 0x1D2CC, 0x1D2EC, 0x1D2EC, 0x246C, 0x246C, 0x2480, 0x2480, + 0x2494, 0x2494, 0x24ED, 0x24ED, 0x16E8D, 0x16E8D, 0x1D2CD, 0x1D2CD, 0x1D2ED, 0x1D2ED, + 0x246D, 0x246D, 0x2481, 0x2481, 0x2495, 0x2495, 0x24EE, 0x24EE, 0x16E8E, 0x16E8E, + 0x1D2CE, 0x1D2CE, 0x1D2EE, 0x1D2EE, 0x246E, 0x246E, 0x2482, 0x2482, 0x2496, 0x2496, + 0x24EF, 0x24EF, 0x16E8F, 0x16E8F, 0x1D2CF, 0x1D2CF, 0x1D2EF, 0x1D2EF, 0x9F9, 0x9F9, + 0x246F, 0x246F, 0x2483, 0x2483, 0x2497, 0x2497, 0x24F0, 0x24F0, 0x16E90, 0x16E90, + 0x1D2D0, 0x1D2D0, 0x1D2F0, 0x1D2F0, 0x16EE, 0x16EE, 0x2470, 0x2470, 0x2484, 0x2484, + 0x2498, 0x2498, 0x24F1, 0x24F1, 0x16E91, 0x16E91, 0x1D2D1, 0x1D2D1, 0x1D2F1, 0x1D2F1, + 0x16EF, 0x16EF, 0x2471, 0x2471, 0x2485, 0x2485, 0x2499, 0x2499, 0x24F2, 0x24F2, + 0x16E92, 0x16E92, 0x1D2D2, 0x1D2D2, 0x1D2F2, 0x1D2F2, 0x16F0, 0x16F0, 0x2472, 0x2472, + 0x2486, 0x2486, 0x249A, 0x249A, 0x24F3, 0x24F3, 0x16E93, 0x16E93, 0x1D2D3, 0x1D2D3, + 0x1D2F3, 0x1D2F3, 0x1373, 0x1373, 0x2473, 0x2473, 0x2487, 0x2487, 0x249B, 0x249B, + 0x24F4, 0x24F4, 0x3039, 0x3039, 0x3249, 0x3249, 0x5344, 0x5344, 0x5EFF, 0x5EFF, + 0x10111, 0x10111, 0x102EB, 0x102EB, 0x103D4, 0x103D4, 0x1085C, 0x1085C, 0x1087F, 0x1087F, + 0x108AE, 0x108AE, 0x108FE, 0x108FE, 0x10918, 0x10918, 0x109CA, 0x109CA, 0x10A45, 0x10A45, + 0x10A9F, 0x10A9F, 0x10AEE, 0x10AEE, 0x10B5D, 0x10B5D, 0x10B7D, 0x10B7D, 0x10BAE, 0x10BAE, + 0x10E6A, 0x10E6A, 0x10F23, 0x10F23, 0x10F53, 0x10F53, 0x10FCA, 0x10FCA, 0x1105C, 0x1105C, + 0x111EB, 0x111EB, 0x1173B, 0x1173B, 0x118EB, 0x118EB, 0x11C64, 0x11C64, 0x1D36A, 0x1D36A, + 0x1EC7B, 0x1EC7B, 0x1ED0B, 0x1ED0B, 0x3251, 0x3251, 0x3252, 0x3252, 0x3253, 0x3253, + 0x3254, 0x3254, 0x3255, 0x3255, 0x3256, 0x3256, 0x3257, 0x3257, 0x3258, 0x3258, + 0x3259, 0x3259, 0x1374, 0x1374, 0x303A, 0x303A, 0x324A, 0x324A, 0x325A, 0x325A, + 0x5345, 0x5345, 0x10112, 0x10112, 0x10165, 0x10165, 0x102EC, 0x102EC, 0x109CB, 0x109CB, + 0x10E6B, 0x10E6B, 0x10F24, 0x10F24, 0x1105D, 0x1105D, 0x111EC, 0x111EC, 0x118EC, 0x118EC, + 0x11C65, 0x11C65, 0x1D36B, 0x1D36B, 0x1EC7C, 0x1EC7C, 0x1ED0C, 0x1ED0C, 0x20983, 0x20983, + 0x325B, 0x325B, 0x325C, 0x325C, 0x325D, 0x325D, 0x325E, 0x325E, 0x325F, 0x325F, + 0x32B1, 0x32B1, 0x32B2, 0x32B2, 0x32B3, 0x32B3, 0x32B4, 0x32B4, 0x1375, 0x1375, + 0x324B, 0x324B, 0x32B5, 0x32B5, 0x534C, 0x534C, 0x10113, 0x10113, 0x102ED, 0x102ED, + 0x109CC, 0x109CC, 0x10E6C, 0x10E6C, 0x1105E, 0x1105E, 0x111ED, 0x111ED, 0x118ED, 0x118ED, + 0x11C66, 0x11C66, 0x12467, 0x12467, 0x1D36C, 0x1D36C, 0x1EC7D, 0x1EC7D, 0x1ED0D, 0x1ED0D, + 0x2098C, 0x2098C, 0x2099C, 0x2099C, 0x32B6, 0x32B6, 0x32B7, 0x32B7, 0x32B8, 0x32B8, + 0x32B9, 0x32B9, 0x32BA, 0x32BA, 0x32BB, 0x32BB, 0x32BC, 0x32BC, 0x32BD, 0x32BD, + 0x32BE, 0x32BE, 0x1376, 0x1376, 0x216C, 0x216C, 0x217C, 0x217C, 0x2186, 0x2186, + 0x324C, 0x324C, 0x32BF, 0x32BF, 0x10114, 0x10114, 0x10144, 0x10144, 0x1014A, 0x1014A, + 0x10151, 0x10151, 0x10166, 0x10169, 0x10174, 0x10174, 0x102EE, 0x102EE, 0x10323, 0x10323, + 0x109CD, 0x109CD, 0x10A7E, 0x10A7E, 0x10CFD, 0x10CFD, 0x10E6D, 0x10E6D, 0x1105F, 0x1105F, + 0x111EE, 0x111EE, 0x118EE, 0x118EE, 0x11C67, 0x11C67, 0x12468, 0x12468, 0x1D36D, 0x1D36D, + 0x1EC7E, 0x1EC7E, 0x1ED0E, 0x1ED0E, 0x1377, 0x1377, 0x324D, 0x324D, 0x10115, 0x10115, + 0x102EF, 0x102EF, 0x109CE, 0x109CE, 0x10E6E, 0x10E6E, 0x11060, 0x11060, 0x111EF, 0x111EF, + 0x118EF, 0x118EF, 0x11C68, 0x11C68, 0x1D36E, 0x1D36E, 0x1EC7F, 0x1EC7F, 0x1ED0F, 0x1ED0F, + 0x1378, 0x1378, 0x324E, 0x324E, 0x10116, 0x10116, 0x102F0, 0x102F0, 0x109CF, 0x109CF, + 0x10E6F, 0x10E6F, 0x11061, 0x11061, 0x111F0, 0x111F0, 0x118F0, 0x118F0, 0x11C69, 0x11C69, + 0x1D36F, 0x1D36F, 0x1EC80, 0x1EC80, 0x1ED10, 0x1ED10, 0x1379, 0x1379, 0x324F, 0x324F, + 0x10117, 0x10117, 0x102F1, 0x102F1, 0x10E70, 0x10E70, 0x11062, 0x11062, 0x111F1, 0x111F1, + 0x118F1, 0x118F1, 0x11C6A, 0x11C6A, 0x1D370, 0x1D370, 0x1EC81, 0x1EC81, 0x1ED11, 0x1ED11, + 0x137A, 0x137A, 0x10118, 0x10118, 0x102F2, 0x102F2, 0x10341, 0x10341, 0x10E71, 0x10E71, + 0x11063, 0x11063, 0x111F2, 0x111F2, 0x118F2, 0x118F2, 0x11C6B, 0x11C6B, 0x1D371, 0x1D371, + 0x1EC82, 0x1EC82, 0x1ED12, 0x1ED12, 0xBF1, 0xBF1, 0xD71, 0xD71, 0x137B, 0x137B, + 0x216D, 0x216D, 0x217D, 0x217D, 0x4F70, 0x4F70, 0x767E, 0x767E, 0x964C, 0x964C, + 0x10119, 0x10119, 0x1014B, 0x1014B, 0x10152, 0x10152, 0x1016A, 0x1016A, 0x102F3, 0x102F3, + 0x103D5, 0x103D5, 0x1085D, 0x1085D, 0x108AF, 0x108AF, 0x108FF, 0x108FF, 0x10919, 0x10919, + 0x109D2, 0x109D2, 0x10A46, 0x10A46, 0x10AEF, 0x10AEF, 0x10B5E, 0x10B5E, 0x10B7E, 0x10B7E, + 0x10BAF, 0x10BAF, 0x10CFE, 0x10CFE, 0x10E72, 0x10E72, 0x10F25, 0x10F25, 0x10F54, 0x10F54, + 0x10FCB, 0x10FCB, 0x11064, 0x11064, 0x111F3, 0x111F3, 0x11C6C, 0x11C6C, 0x16B5C, 0x16B5C, + 0x1EC83, 0x1EC83, 0x1ED13, 0x1ED13, 0x7695, 0x7695, 0x1011A, 0x1011A, 0x102F4, 0x102F4, + 0x109D3, 0x109D3, 0x10E73, 0x10E73, 0x1EC84, 0x1EC84, 0x1ED14, 0x1ED14, 0x1011B, 0x1011B, + 0x1016B, 0x1016B, 0x102F5, 0x102F5, 0x109D4, 0x109D4, 0x10E74, 0x10E74, 0x1EC85, 0x1EC85, + 0x1ED15, 0x1ED15, 0x1011C, 0x1011C, 0x102F6, 0x102F6, 0x109D5, 0x109D5, 0x10E75, 0x10E75, + 0x1EC86, 0x1EC86, 0x1ED16, 0x1ED16, 0x1ED38, 0x1ED38, 0x216E, 0x216E, 0x217E, 0x217E, + 0x1011D, 0x1011D, 0x10145, 0x10145, 0x1014C, 0x1014C, 0x10153, 0x10153, 0x1016C, 0x10170, + 0x102F7, 0x102F7, 0x109D6, 0x109D6, 0x10E76, 0x10E76, 0x1EC87, 0x1EC87, 0x1ED17, 0x1ED17, + 0x1011E, 0x1011E, 0x102F8, 0x102F8, 0x109D7, 0x109D7, 0x10E77, 0x10E77, 0x1EC88, 0x1EC88, + 0x1ED18, 0x1ED18, 0x1ED39, 0x1ED39, 0x1011F, 0x1011F, 0x102F9, 0x102F9, 0x109D8, 0x109D8, + 0x10E78, 0x10E78, 0x1EC89, 0x1EC89, 0x1ED19, 0x1ED19, 0x10120, 0x10120, 0x102FA, 0x102FA, + 0x109D9, 0x109D9, 0x10E79, 0x10E79, 0x1EC8A, 0x1EC8A, 0x1ED1A, 0x1ED1A, 0x10121, 0x10121, + 0x102FB, 0x102FB, 0x1034A, 0x1034A, 0x109DA, 0x109DA, 0x10E7A, 0x10E7A, 0x1EC8B, 0x1EC8B, + 0x1ED1B, 0x1ED1B, 0xBF2, 0xBF2, 0xD72, 0xD72, 0x216F, 0x216F, 0x217F, 0x2180, + 0x4EDF, 0x4EDF, 0x5343, 0x5343, 0x9621, 0x9621, 0x10122, 0x10122, 0x1014D, 0x1014D, + 0x10154, 0x10154, 0x10171, 0x10171, 0x1085E, 0x1085E, 0x109DB, 0x109DB, 0x10A47, 0x10A47, + 0x10B5F, 0x10B5F, 0x10B7F, 0x10B7F, 0x10CFF, 0x10CFF, 0x11065, 0x11065, 0x111F4, 0x111F4, + 0x1EC8C, 0x1EC8C, 0x1ED1C, 0x1ED1C, 0x10123, 0x10123, 0x109DC, 0x109DC, 0x1EC8D, 0x1EC8D, + 0x1ED1D, 0x1ED1D, 0x1ED3A, 0x1ED3A, 0x10124, 0x10124, 0x109DD, 0x109DD, 0x1EC8E, 0x1EC8E, + 0x1ED1E, 0x1ED1E, 0x10125, 0x10125, 0x109DE, 0x109DE, 0x1EC8F, 0x1EC8F, 0x1ED1F, 0x1ED1F, + 0x2181, 0x2181, 0x10126, 0x10126, 0x10146, 0x10146, 0x1014E, 0x1014E, 0x10172, 0x10172, + 0x109DF, 0x109DF, 0x1EC90, 0x1EC90, 0x1ED20, 0x1ED20, 0x10127, 0x10127, 0x109E0, 0x109E0, + 0x1EC91, 0x1EC91, 0x1ED21, 0x1ED21, 0x10128, 0x10128, 0x109E1, 0x109E1, 0x1EC92, 0x1EC92, + 0x1ED22, 0x1ED22, 0x10129, 0x10129, 0x109E2, 0x109E2, 0x1EC93, 0x1EC93, 0x1ED23, 0x1ED23, + 0x1012A, 0x1012A, 0x109E3, 0x109E3, 0x1EC94, 0x1EC94, 0x1ED24, 0x1ED24, 0x137C, 0x137C, + 0x2182, 0x2182, 0x4E07, 0x4E07, 0x842C, 0x842C, 0x1012B, 0x1012B, 0x10155, 0x10155, + 0x1085F, 0x1085F, 0x109E4, 0x109E4, 0x16B5D, 0x16B5D, 0x1EC95, 0x1EC95, 0x1ECB3, 0x1ECB3, + 0x1ED25, 0x1ED25, 0x1ED3B, 0x1ED3B, 0x1012C, 0x1012C, 0x109E5, 0x109E5, 0x1EC96, 0x1EC96, + 0x1ED26, 0x1ED26, 0x1012D, 0x1012D, 0x109E6, 0x109E6, 0x1EC97, 0x1EC97, 0x1ED27, 0x1ED27, + 0x1012E, 0x1012E, 0x109E7, 0x109E7, 0x1EC98, 0x1EC98, 0x1ED28, 0x1ED28, 0x2187, 0x2187, + 0x1012F, 0x1012F, 0x10147, 0x10147, 0x10156, 0x10156, 0x109E8, 0x109E8, 0x1EC99, 0x1EC99, + 0x1ED29, 0x1ED29, 0x10130, 0x10130, 0x109E9, 0x109E9, 0x1EC9A, 0x1EC9A, 0x1ED2A, 0x1ED2A, + 0x10131, 0x10131, 0x109EA, 0x109EA, 0x1EC9B, 0x1EC9B, 0x1ED2B, 0x1ED2B, 0x10132, 0x10132, + 0x109EB, 0x109EB, 0x1EC9C, 0x1EC9C, 0x1ED2C, 0x1ED2C, 0x10133, 0x10133, 0x109EC, 0x109EC, + 0x1EC9D, 0x1EC9D, 0x1ED2D, 0x1ED2D, 0x2188, 0x2188, 0x109ED, 0x109ED, 0x1EC9E, 0x1EC9E, + 0x1ECA0, 0x1ECA0, 0x1ECB4, 0x1ECB4, 0x109EE, 0x109EE, 0x1EC9F, 0x1EC9F, 0x12432, 0x12432, + 0x109EF, 0x109EF, 0x109F0, 0x109F0, 0x12433, 0x12433, 0x109F1, 0x109F1, 0x109F2, 0x109F2, + 0x109F3, 0x109F3, 0x109F4, 0x109F4, 0x109F5, 0x109F5, 0x16B5E, 0x16B5E, 0x1ECA1, 0x1ECA1, + 0x1ECA2, 0x1ECA2, 0x4EBF, 0x4EBF, 0x5104, 0x5104, 0x16B5F, 0x16B5F, 0x79ED, 0x79ED, + 0x16B60, 0x16B60, 0x5146, 0x5146, 0x16B61, 0x16B61, 0x4EAC, 0x4EAC, + }; + + private static final MathContext PERL_NUMERIC_CONTEXT = + new MathContext(4, RoundingMode.HALF_EVEN); + private static final UnicodeSet[] SETS = buildSets(); + private static final BigDecimal[] DECIMALS = buildDecimals(); + private static final BigDecimal[] ALTERNATE_DECIMALS = buildAlternateDecimals(); + private static final UnicodeSet ASSIGNED = buildAssignedSet(); + private static final UnicodeSet NAN = new UnicodeSet(0, 0x10ffff) + .removeAll(ASSIGNED).freeze(); + + static int valueCount() { + return CANONICAL_VALUES.length; + } + + static String canonicalValue(int index) { + return CANONICAL_VALUES[index]; + } + + static long numerator(int index) { + return NUMERATORS[index]; + } + + static short valueForDecimal(BigDecimal decimal) { + double binaryValue = decimal.doubleValue(); + if (!Double.isFinite(binaryValue)) return INVALID; + BigDecimal canonical = new BigDecimal(binaryValue).round(PERL_NUMERIC_CONTEXT); + for (short index = 0; index < DECIMALS.length; index++) { + if (DENOMINATORS[index] != 1 + && (DECIMALS[index].compareTo(canonical) == 0 + || ALTERNATE_DECIMALS[index].compareTo(canonical) == 0)) { + return index; + } + } + return INVALID; + } + + static int denominator(int index) { + return DENOMINATORS[index]; + } + + static UnicodeSet set(int index) { + return SETS[index]; + } + + static UnicodeSet nanSet() { + return NAN; + } + + static UnicodeSet assignedSet() { + return ASSIGNED; + } + + static short valueForRational(long numerator, long denominator) { + if (denominator == 0 || numerator == Long.MIN_VALUE || denominator == Long.MIN_VALUE) { + return INVALID; + } + if (denominator < 0) { + numerator = -numerator; + denominator = -denominator; + } + long divisor = gcd(numerator, denominator); + numerator /= divisor; + denominator /= divisor; + for (short index = 0; index < CANONICAL_VALUES.length; index++) { + if (NUMERATORS[index] == numerator && DENOMINATORS[index] == denominator) { + return index; + } + } + return INVALID; + } + + static boolean isPropertyAlias(String alias) { + boolean hasIsPrefix = alias != null && alias.startsWith("Is"); + String normalized = loose(hasIsPrefix ? alias.substring(2) : alias); + switch (normalized) { + case "numericvalue": + case "nv": + return true; + default: + return false; + } + } + + private static UnicodeSet[] buildSets() { + UnicodeSet[] sets = new UnicodeSet[CANONICAL_VALUES.length]; + for (int value = 0; value < sets.length; value++) { + UnicodeSet set = new UnicodeSet(); + for (int range = RANGE_OFFSETS[value]; range < RANGE_OFFSETS[value + 1]; range++) { + set.add(RANGE_ENDPOINTS[range * 2], RANGE_ENDPOINTS[range * 2 + 1]); + } + sets[value] = set.freeze(); + } + return sets; + } + + private static BigDecimal[] buildDecimals() { + BigDecimal[] decimals = new BigDecimal[DECIMAL_ALIASES.length]; + for (int index = 0; index < decimals.length; index++) { + decimals[index] = new BigDecimal(DECIMAL_ALIASES[index]); + if (index > 0 && decimals[index - 1].compareTo(decimals[index]) >= 0) { + throw new IllegalStateException("Numeric_Value decimals are not sorted"); + } + } + return decimals; + } + + private static BigDecimal[] buildAlternateDecimals() { + MathContext alternateContext = new MathContext(4, RoundingMode.HALF_UP); + BigDecimal[] decimals = new BigDecimal[NUMERATORS.length]; + for (int index = 0; index < decimals.length; index++) { + decimals[index] = BigDecimal.valueOf(NUMERATORS[index]) + .divide(BigDecimal.valueOf(DENOMINATORS[index]), alternateContext); + } + return decimals; + } + + private static UnicodeSet buildAssignedSet() { + UnicodeSet assigned = new UnicodeSet(); + for (UnicodeSet set : SETS) assigned.addAll(set); + return assigned.freeze(); + } + + private static long gcd(long left, long right) { + left = Math.abs(left); + while (right != 0) { + long remainder = left % right; + left = right; + right = remainder; + } + return left; + } + + private static String loose(String alias) { + if (alias == null) return ""; + StringBuilder normalized = new StringBuilder(alias.length()); + for (int i = 0; i < alias.length(); i++) { + char character = alias.charAt(i); + if (character == '_' || character == '-' || character == ' ' + || (character >= '\t' && character <= '\r')) continue; + normalized.append(Character.toLowerCase(character)); + } + return normalized.toString(); + } + + private PerlUnicodeNumericValueData() { + } +} diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index 0641292fa5..32c9072324 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -6,6 +6,8 @@ import org.perlonjava.app.scriptengine.PerlLanguageProvider; import org.perlonjava.runtime.runtimetypes.*; +import java.math.BigDecimal; +import java.math.BigInteger; import java.util.ArrayList; import java.util.Collections; import java.util.LinkedHashSet; @@ -695,7 +697,13 @@ private static String translateUnicodeProperty(String property, boolean negated, property = property.substring(1).trim(); negated = !negated; } + boolean isPrefixedNumericWildcard = + isPerlIsPrefixedNumericWildcard(property); property = normalizePerlIsPropertyAssignment(property); + if (isPrefixedNumericWildcard) { + throw new IllegalArgumentException( + "Is-prefixed Numeric_Value properties do not accept wildcard values"); + } if (property.startsWith("utf8::")) { String userPropertyName = property.substring("utf8::".length()); if (!userPropertyName.matches("[A-Za-z_][A-Za-z0-9_]*")) { @@ -997,6 +1005,18 @@ && isCanonicalCombiningClassProperty(alias.substring(0, assignment))) { } return eastAsianWidth; } + if (assignment > 0 && assignment < alias.length() - 1 + && PerlUnicodeNumericValueData.isPropertyAlias( + alias.substring(0, assignment))) { + UnicodeSet numericValue = resolvePerlNumericValue( + alias.substring(assignment + 1)); + if (numericValue == null) { + throw new IllegalArgumentException( + "Unsupported Numeric_Value value: " + + alias.substring(assignment + 1).trim()); + } + return numericValue; + } if (assignment > 0 && assignment < alias.length() - 1) { Boolean value = perlBooleanPropertyValue(alias.substring(assignment + 1)); if (value != null) { @@ -1056,6 +1076,139 @@ private static Boolean perlBooleanPropertyValue(String value) { }; } + private static boolean isPerlIsPrefixedNumericWildcard(String property) { + int assignment = propertyValueDelimiter(property); + if (assignment <= 0 || assignment == property.length() - 1) return false; + String name = property.substring(0, assignment); + return name.length() > 2 + && name.charAt(0) == 'I' + && name.charAt(1) == 's' + && PerlUnicodeNumericValueData.isPropertyAlias(name) + && perlNumericWildcardBody(property.substring(assignment + 1)) != null; + } + + private static UnicodeSet resolvePerlNumericValue(String value) { + String wildcard = perlNumericWildcardBody(value); + if (wildcard != null) { + Pattern valuePattern; + try { + valuePattern = Pattern.compile(wildcard); + } catch (RuntimeException invalidPattern) { + return null; + } + UnicodeSet result = new UnicodeSet(); + for (int index = 0; index < PerlUnicodeNumericValueData.valueCount(); index++) { + if (valuePattern.matcher( + PerlUnicodeNumericValueData.canonicalValue(index)).matches()) { + result.addAll(PerlUnicodeNumericValueData.set(index)); + } + } + if (valuePattern.matcher("NaN").matches() + || valuePattern.matcher("nan").matches()) { + result.addAll(PerlUnicodeNumericValueData.nanSet()); + } + return result.isEmpty() ? null : result.freeze(); + } + + if (loosePropertyName(value).equals("nan")) { + return PerlUnicodeNumericValueData.nanSet(); + } + + short index = perlNumericValueIndex(value); + return index == PerlUnicodeNumericValueData.INVALID + ? null + : PerlUnicodeNumericValueData.set(index); + } + + private static String perlNumericWildcardBody(String value) { + String trimmed = value.trim(); + if (trimmed.startsWith("/\\A") && trimmed.endsWith("\\z/") + && trimmed.length() > 6) { + return trimmed.substring(3, trimmed.length() - 3); + } + if (trimmed.startsWith(":\\A") && trimmed.endsWith("\\z:") + && trimmed.length() > 6) { + return trimmed.substring(3, trimmed.length() - 3); + } + return null; + } + + private static short perlNumericValueIndex(String value) { + try { + if (value.indexOf('/') >= 0) { + return perlRationalValueIndex(value); + } + + String normalized = removePerlNumericLooseCharacters(value); + if (!normalized.matches("(?:\\+?[0-9]+(?:\\.[0-9]*)?" + + "|-[0-9]*(?:\\.[0-9]+)?)(?:[eE][+-]?[0-9]+)?")) { + return PerlUnicodeNumericValueData.INVALID; + } + BigDecimal decimal = new BigDecimal(normalized); + BigInteger numerator = decimal.unscaledValue(); + BigInteger denominator = BigInteger.ONE; + if (decimal.scale() > 0) { + denominator = BigInteger.TEN.pow(decimal.scale()); + } else if (decimal.scale() < 0) { + numerator = numerator.multiply(BigInteger.TEN.pow(-decimal.scale())); + } + short exact = rationalValueIndex(numerator, denominator); + return exact != PerlUnicodeNumericValueData.INVALID + ? exact + : PerlUnicodeNumericValueData.valueForDecimal(decimal); + } catch (ArithmeticException | NumberFormatException invalidValue) { + return PerlUnicodeNumericValueData.INVALID; + } + } + + private static short perlRationalValueIndex(String value) { + value = value.trim(); + for (int i = 0; i < value.length(); i++) { + char character = value.charAt(i); + if (character == ' ' || character >= '\t' && character <= '\r') { + return PerlUnicodeNumericValueData.INVALID; + } + } + String[] parts = value.split("/", -1); + if (parts.length != 2 || parts[0].endsWith("_")) { + return PerlUnicodeNumericValueData.INVALID; + } + String numeratorText = parts[0].replace("_", ""); + String denominatorText = parts[1]; + if (denominatorText.startsWith("+")) denominatorText = denominatorText.substring(1); + while (denominatorText.startsWith("_")) denominatorText = denominatorText.substring(1); + denominatorText = denominatorText.replace("_", ""); + if (!numeratorText.matches("[+-]?[0-9]+") + || !denominatorText.matches("[0-9]+")) { + return PerlUnicodeNumericValueData.INVALID; + } + return rationalValueIndex( + new BigInteger(numeratorText), new BigInteger(denominatorText)); + } + + private static short rationalValueIndex(BigInteger numerator, BigInteger denominator) { + if (denominator.signum() == 0) return PerlUnicodeNumericValueData.INVALID; + BigInteger divisor = numerator.gcd(denominator); + numerator = numerator.divide(divisor); + denominator = denominator.divide(divisor); + if (numerator.bitLength() > 63 || denominator.bitLength() > 63) { + return PerlUnicodeNumericValueData.INVALID; + } + return PerlUnicodeNumericValueData.valueForRational( + numerator.longValueExact(), denominator.longValueExact()); + } + + private static String removePerlNumericLooseCharacters(String value) { + StringBuilder normalized = new StringBuilder(value.length()); + for (int i = 0; i < value.length(); i++) { + char character = value.charAt(i); + if (character == '_' || character == ' ' + || character >= '\t' && character <= '\r') continue; + normalized.append(character); + } + return normalized.toString(); + } + private static boolean isGeneralCategoryProperty(String property) { return switch (loosePropertyName(property)) { case "gc", "generalcategory", "category" -> true; diff --git a/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeNumericValueDataTest.java b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeNumericValueDataTest.java new file mode 100644 index 0000000000..7dc2ec980d --- /dev/null +++ b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeNumericValueDataTest.java @@ -0,0 +1,149 @@ +package org.perlonjava.runtime.regex; + +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.math.BigDecimal; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.concurrent.TimeUnit; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; + +@Tag("unit") +class PerlUnicodeNumericValueDataTest { + private static final int MAX_CODE_POINT = 0x10ffff; + + @Test + void usesPinnedPerl544Unicode17Data() { + assertEquals("17.0.0", PerlUnicodeNumericValueData.UNICODE_VERSION); + assertEquals(144, PerlUnicodeNumericValueData.valueCount()); + + assertTrue(set("-1/2").contains(0x0f33)); + assertTrue(set("1/2").contains(0x00bd)); + assertTrue(set("0").contains('0')); + assertTrue(set("10000000000000000").contains(0x4eac)); + assertTrue(set("3/2").contains(0x16ff5)); + assertFalse(set("3/2").contains(0x16ff6)); + assertTrue(set("2").contains(0x16ff6)); + assertTrue(PerlUnicodeNumericValueData.nanSet().contains('A')); + assertTrue(PerlUnicodeNumericValueData.nanSet().contains(MAX_CODE_POINT)); + } + + @Test + void formsACompleteDisjointPartition() { + var union = new com.ibm.icu.text.UnicodeSet(); + int rangeCount = 0; + for (int index = 0; index < PerlUnicodeNumericValueData.valueCount(); index++) { + var set = PerlUnicodeNumericValueData.set(index); + assertTrue(set.isFrozen(), PerlUnicodeNumericValueData.canonicalValue(index)); + assertFalse(set.isEmpty(), PerlUnicodeNumericValueData.canonicalValue(index)); + assertTrue(new com.ibm.icu.text.UnicodeSet(union).retainAll(set).isEmpty(), + PerlUnicodeNumericValueData.canonicalValue(index)); + union.addAll(set); + rangeCount += set.getRangeCount(); + } + + assertEquals(1_979, rangeCount); + assertEquals(2_023, union.size()); + assertEquals(PerlUnicodeNumericValueData.assignedSet(), union); + assertTrue(PerlUnicodeNumericValueData.assignedSet().isFrozen()); + assertTrue(PerlUnicodeNumericValueData.nanSet().isFrozen()); + assertTrue(new com.ibm.icu.text.UnicodeSet(union) + .retainAll(PerlUnicodeNumericValueData.nanSet()).isEmpty()); + assertEquals(1_112_089, PerlUnicodeNumericValueData.nanSet().size()); + union.addAll(PerlUnicodeNumericValueData.nanSet()); + assertEquals(new com.ibm.icu.text.UnicodeSet(0, MAX_CODE_POINT), union); + } + + @Test + void exposesReducedExactRationals() { + for (short index = 0; index < PerlUnicodeNumericValueData.valueCount(); index++) { + long numerator = PerlUnicodeNumericValueData.numerator(index); + int denominator = PerlUnicodeNumericValueData.denominator(index); + String expected = denominator == 1 + ? Long.toString(numerator) + : numerator + "/" + denominator; + assertEquals(expected, PerlUnicodeNumericValueData.canonicalValue(index)); + assertEquals(index, + PerlUnicodeNumericValueData.valueForRational(numerator, denominator)); + } + + assertEquals(index("1/2"), PerlUnicodeNumericValueData.valueForRational(2, 4)); + assertEquals(index("-1/2"), PerlUnicodeNumericValueData.valueForRational(1, -2)); + assertEquals(index("1/2"), PerlUnicodeNumericValueData.valueForRational(-1, -2)); + assertEquals(PerlUnicodeNumericValueData.INVALID, + PerlUnicodeNumericValueData.valueForRational(1, 0)); + assertEquals(PerlUnicodeNumericValueData.INVALID, + PerlUnicodeNumericValueData.valueForRational(7, 11)); + assertEquals(PerlUnicodeNumericValueData.INVALID, + PerlUnicodeNumericValueData.valueForRational(Long.MIN_VALUE, 1)); + } + + @Test + void exposesPinnedPerlDecimalKeywordAliases() { + assertEquals(index("1/12"), + PerlUnicodeNumericValueData.valueForDecimal(new BigDecimal("8.333e-02"))); + assertEquals(index("1/64"), + PerlUnicodeNumericValueData.valueForDecimal(new BigDecimal("0.01562"))); + assertEquals(index("1/64"), + PerlUnicodeNumericValueData.valueForDecimal(new BigDecimal("0.01563"))); + assertEquals(index("1/6"), + PerlUnicodeNumericValueData.valueForDecimal(new BigDecimal("0.1667"))); + assertEquals(index("3/64"), + PerlUnicodeNumericValueData.valueForDecimal(new BigDecimal("4.688e-02"))); + assertEquals(PerlUnicodeNumericValueData.INVALID, + PerlUnicodeNumericValueData.valueForDecimal(new BigDecimal("0.1668"))); + assertEquals(index("1/7"), + PerlUnicodeNumericValueData.valueForDecimal(new BigDecimal("0.14295"))); + assertEquals(PerlUnicodeNumericValueData.INVALID, + PerlUnicodeNumericValueData.valueForDecimal(new BigDecimal("0.1428"))); + } + + @Test + void followsPerlLoosePropertyAliasRules() { + assertTrue(PerlUnicodeNumericValueData.isPropertyAlias("nv")); + assertTrue(PerlUnicodeNumericValueData.isPropertyAlias("NV")); + assertTrue(PerlUnicodeNumericValueData.isPropertyAlias("Numeric_Value")); + assertTrue(PerlUnicodeNumericValueData.isPropertyAlias("numeric-value")); + assertTrue(PerlUnicodeNumericValueData.isPropertyAlias("numeric value")); + assertTrue(PerlUnicodeNumericValueData.isPropertyAlias("Is_Nv")); + assertTrue(PerlUnicodeNumericValueData.isPropertyAlias("IsNumericValue")); + + assertFalse(PerlUnicodeNumericValueData.isPropertyAlias("is_Nv")); + assertFalse(PerlUnicodeNumericValueData.isPropertyAlias("IS_NV")); + assertFalse(PerlUnicodeNumericValueData.isPropertyAlias("Numeric\u1680Value")); + assertFalse(PerlUnicodeNumericValueData.isPropertyAlias("value")); + assertFalse(PerlUnicodeNumericValueData.isPropertyAlias(null)); + } + + @Test + void generatorReproducesCheckedInDataByteForByte() throws Exception { + Path generator = Path.of("dev", "tools", "generate_perl_unicode_numeric_value_data.pl"); + Path generated = Path.of("src", "main", "java", "org", "perlonjava", "runtime", + "regex", "PerlUnicodeNumericValueData.java"); + Process process = new ProcessBuilder("perl", generator.toString()) + .redirectErrorStream(true) + .start(); + byte[] actual = process.getInputStream().readAllBytes(); + assertTrue(process.waitFor(30, TimeUnit.SECONDS), "generator timed out"); + assertEquals(0, process.exitValue(), new String(actual, StandardCharsets.UTF_8)); + assertArrayEquals(Files.readAllBytes(generated), actual); + } + + private static com.ibm.icu.text.UnicodeSet set(String canonicalValue) { + return PerlUnicodeNumericValueData.set(index(canonicalValue)); + } + + private static short index(String canonicalValue) { + for (short index = 0; index < PerlUnicodeNumericValueData.valueCount(); index++) { + if (canonicalValue.equals(PerlUnicodeNumericValueData.canonicalValue(index))) { + return index; + } + } + throw new AssertionError("Unknown Numeric_Value " + canonicalValue); + } +} diff --git a/src/test/resources/unit/regex/unicode_numeric_value_data.t b/src/test/resources/unit/regex/unicode_numeric_value_data.t new file mode 100644 index 0000000000..0aaeb6f051 --- /dev/null +++ b/src/test/resources/unit/regex/unicode_numeric_value_data.t @@ -0,0 +1,76 @@ +use strict; +use warnings; +use utf8; +use Test::More; +use Unicode::UCD (); + +sub compile_property { + my ($expression) = @_; + local $SIG{__WARN__} = sub { }; + return eval "qr!\\p{$expression}!"; +} + +sub matches_property { + my ($expression, $code_point) = @_; + my $regex = compile_property($expression); + return defined($regex) && chr($code_point) =~ $regex; +} + + ok(matches_property('nv=1', 0x0031), 'short property name and integer value'); + ok(matches_property('Numeric_Value:1', 0x0031), 'long property name and colon'); + ok(matches_property('nU-mErIc value=+ 000_001', 0x0031), + 'property and integer spellings use Perl loose matching'); + ok(matches_property('Is_NV=1', 0x0031), 'short exact-case Is property form'); + ok(matches_property('Is Numeric Value=1', 0x0031), + 'long exact-case Is property form is loose after Is'); + ok(!defined(compile_property('is_nv=1')), 'Is property prefix remains case-sensitive'); + + for my $value ('1/2', '+1/2', '01/02', '10/20', '3/6', '1/_2', '1_0/2_0', + '1/+2', '0.5', '00.500', '5e-1') { + ok(matches_property("nv=$value", 0x00BD), "half spelling $value"); + } + for my $value ('-1/2', '-01/02', '-0.5', '-.5', '-5e-1') { + ok(matches_property("nv=$value", 0x0F33), "negative half spelling $value"); + } + ok(matches_property('Numeric_Value: -0000001/0000002', 0x0F33), + 'leading loose whitespace is accepted before a rational value'); + ok(matches_property('nv=8.333e-02', 0x109F6), + 'the pinned decimal alias matches one twelfth'); + ok(matches_property('nv=0.1667', 0x2159), + 'the pinned rounded decimal alias matches one sixth'); + ok(!defined(compile_property('nv=0.1668')), + 'a nearby decimal outside the pinned aliases is rejected'); + ok(matches_property('nv=0.14295', 0x2150), + 'NV canonicalization follows binary floating-point rounding'); + ok(!defined(compile_property('nv=0.1428')), + 'NV canonicalization rejects the adjacent rounded value'); + + ok(matches_property('nv=10000000000000000', 0x4EAC), + 'maximum Unicode numeric value fits an exact integer'); + ok(matches_property('nv=NaN', 0x0041), 'NaN includes assigned nonnumeric code points'); + ok(matches_property('nv=N_A_N', 0x0378), 'NaN alias is loose and includes unassigned code points'); + ok(matches_property('nv=nan', 0x10FFFF), 'NaN includes the Unicode maximum'); + ok(!matches_property('nv=NaN', 0x0031), 'NaN excludes numeric code points'); + + for my $value ('.5', '1 / 2', '1_/2', '1/_/2', '1.0/2.0', '1/-2', '0/0', + '1/0', 'None', 'Not_A_Number') { + ok(!defined(compile_property("nv=$value")), "invalid numeric spelling $value"); + } + + ok(matches_property('nv=/\A1\/2\z/', 0x00BD), + 'wildcard value matches the canonical rational spelling'); + ok(matches_property('nv=:\A300\z:', 0x1011B), + 'alternate wildcard delimiter matches an integer value'); + ok(matches_property('nv=:\Anan\z:', 0x0041), + 'wildcard value matches the lowercase NaN alias'); + ok(!defined(compile_property('Is_Nv=/\A1\z/')), + 'Is-prefixed property names reject wildcard values'); + ok(!defined(compile_property('nv=*')), 'bare wildcard marker is rejected'); + + SKIP: { + skip 'requires Unicode 17 Numeric_Value data', 2 + if Unicode::UCD::UnicodeVersion() lt '17.0.0'; + ok(matches_property('nv=3/2', 0x16FF5), 'Unicode 17 fractional-value boundary'); + ok(!matches_property('nv=3/2', 0x16FF6), 'Unicode 17 adjacent nonmatch boundary'); + } +done_testing; From 4d73b3a1b5fd1211acf45a3cd9f3b623253c7b0f Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:53 +0200 Subject: [PATCH 33/60] feat(regex): add pinned Joining_Group parity (#1038) Squashed integration of PR #1038 at eb614d954. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 32 +- ...enerate_perl_unicode_joining_group_data.pl | 279 ++++++++++++++++++ docs/reference/feature-matrix.md | 4 +- .../regex/PerlUnicodeJoiningGroupData.java | 181 ++++++++++++ .../runtime/regex/UnicodeResolver.java | 82 ++++- .../PerlUnicodeJoiningGroupDataTest.java | 102 +++++++ .../unit/regex/unicode_joining_group_pinned.t | 92 ++++++ 7 files changed, 757 insertions(+), 15 deletions(-) create mode 100644 dev/tools/generate_perl_unicode_joining_group_data.pl create mode 100644 src/main/java/org/perlonjava/runtime/regex/PerlUnicodeJoiningGroupData.java create mode 100644 src/test/java/org/perlonjava/runtime/regex/PerlUnicodeJoiningGroupDataTest.java create mode 100644 src/test/resources/unit/regex/unicode_joining_group_pinned.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 3f855e0db6..3df6dcabb3 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,7 +224,7 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1036, stacked +The published integration stack is preserved through draft PR #1037, stacked on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the @@ -238,10 +238,10 @@ line boundaries. PR #1031 adds pinned Canonical_Combining_Class sets and valid empty-property rendering. PR #1032 integrates native numeric escapes through U+10FFFF; PR #1033 adds pinned Bidi_Class sets; PR #1034 integrates native vertical-whitespace escapes; PR #1035 adds Decomposition_Type and PR #1036 -adds East_Asian_Width. The current WIP integrates Numeric_Value; independently -validated Joining_Group, Block, Script/Script_Extensions, and break-property -value data are ready, while generic and specialized binary-property slices -advance independently. +adds East_Asian_Width and PR #1037 adds Numeric_Value. The current WIP +integrates Joining_Group; independently validated Block, +Script/Script_Extensions, break-property values, and generic and specialized +binary-property data are ready for focused resolver integration. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -361,6 +361,16 @@ per backend. Chunks 02–03 gain 13,976 assertions with zero numbered regression and exact JVM/interpreter success sets, raising current generated evidence to 356,848/407,367. +`Joining_Group`/`jg` assignments now resolve all 106 values from a complete +pinned Unicode 17 partition. Loose aliases, the ordered `No_Joining_Group` +default, the alternate `Hamza_On_Heh_Goal` wildcard name, canonical and +squeezed wildcard values, exact case-sensitive `Is` policy, and wildcard +diagnostics follow Perl 5.44. The focused oracle passes 49/49 on system Perl, +JVM, and interpreter; protected boundary smoke remains 169/169 per backend. +Chunks 01–04 gain 4,290 assertions with no pass-count regression and exact +JVM/interpreter success sets, raising current generated evidence to +361,138/407,367. + Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while ordinary and substitution callbacks execute, without misclassifying escaped or @@ -626,6 +636,10 @@ is retained for now. generated decimal keyword aliases, loose forms, and wildcard policy. The focused oracle passes 50/50; chunks 02–03 gain 13,976 assertions with zero numbered regressions and exact backend identity. + - [x] Generated and integrated a complete pinned Unicode 17.0 Joining_Group + partition with loose aliases, ordered defaults, alternate wildcard names, + and exact `Is`/wildcard rejection policy. The focused oracle passes 49/49; + chunks 01–04 gain 4,290 assertions with exact backend identity. - [x] Integrated native Perl `\v`/`\V` dispatch inside and outside character classes (`1eff1db97`, integrated as `6328935cd`). The focused oracle passes 92/92 and unchanged `reg_posixcc.t` passes 2,560/2,560 on both backends. @@ -671,14 +685,14 @@ is retained for now. ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1036. Publish the validated - Numeric_Value integration in a separate focused WIP PR, preserving the +1. Land review-ready PR #1024 and draft PRs #1025–#1037. Publish the validated + Joining_Group integration in a separate focused WIP PR, preserving the independently reviewed data commit. 2. Preserve the now-complete native Joni boundary corpus at 239,866/239,866: sentence chunk 05, line chunks 06–09, and word chunk 10 must remain exact on JVM and interpreter while property and parser work continues. -3. Integrate the independently generated Joining_Group, Block, - Script/Script_Extensions, and break-property value slices, then the generic +3. Integrate the independently generated Block, Script/Script_Extensions, and + break-property value slices, then the generic and specialized binary-property families currently advancing in parallel. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. diff --git a/dev/tools/generate_perl_unicode_joining_group_data.pl b/dev/tools/generate_perl_unicode_joining_group_data.pl new file mode 100644 index 0000000000..6cbe596d42 --- /dev/null +++ b/dev/tools/generate_perl_unicode_joining_group_data.pl @@ -0,0 +1,279 @@ +#!/usr/bin/env perl +use strict; +use warnings; +use Digest::SHA qw(sha256_hex); +use File::Spec; +use FindBin; + +my $expected_version = '17.0.0'; +my $root = File::Spec->catdir($FindBin::Bin, '..', '..'); +my $unicore = File::Spec->catdir($root, 'perl5', 'lib', 'unicore'); +my @sources = ( + { + name => 'DerivedJoiningGroup-17.0.0.txt', + path => File::Spec->catfile($unicore, 'extracted', 'DJoinGroup.txt'), + hash => 'bb67e0c00b88acfa5be633967b66b23326844a86e49c6fde7b57960d3af66cae', + version => qr/^# DerivedJoiningGroup-\Q$expected_version\E\.txt$/m, + }, + { + name => 'PropertyValueAliases-17.0.0.txt', + path => File::Spec->catfile($unicore, 'PropValueAliases.txt'), + hash => '670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01', + version => qr/^# PropertyValueAliases-\Q$expected_version\E\.txt$/m, + }, + { + name => 'PropertyAliases-17.0.0.txt', + path => File::Spec->catfile($unicore, 'PropertyAliases.txt'), + hash => '4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb', + version => qr/^# PropertyAliases-\Q$expected_version\E\.txt$/m, + }, +); + +sub read_source { + my ($source) = @_; + open my $input, '<:raw', $source->{path} + or die "Cannot read $source->{path}: $!\n"; + local $/; + my $text = <$input>; + close $input or die "Cannot close $source->{path}: $!\n"; + my $actual_hash = sha256_hex($text); + die "$source->{path} SHA-256 mismatch: expected $source->{hash}, found $actual_hash\n" + unless $actual_hash eq $source->{hash}; + die "$source->{path} is not pinned Unicode $expected_version data\n" + unless $text =~ $source->{version}; + $source->{text} = $text; +} + +sub trim { + my ($text) = @_; + $text =~ s/^\s+|\s+$//g; + return $text; +} + +sub loose_name { + my ($name) = @_; + $name = lc $name; + $name =~ s/[\s_-]+//g; + return $name; +} + +sub parse_range { + my ($text) = @_; + my ($start, $end) = split /\.\./, $text; + return (hex($start), hex(defined $end ? $end : $start)); +} + +read_source($_) for @sources; + +my $version_path = File::Spec->catfile($unicore, 'version'); +open my $version_input, '<', $version_path or die "Cannot read $version_path: $!\n"; +chomp(my $unicode_version = <$version_input>); +close $version_input or die "Cannot close $version_path: $!\n"; +die "Expected Unicode $expected_version, found $unicode_version\n" + unless $unicode_version eq $expected_version; + +my (@short_values, @long_values, %alias_index, %wildcard_value_index); +for my $line (split /\n/, $sources[1]{text}) { + next if $line =~ /^\s*#/; + $line =~ s/#.*$//; + my @fields = map { trim($_) } split /;/, $line; + next unless @fields >= 3 && $fields[0] eq 'jg'; + my $index = scalar @short_values; + push @short_values, $fields[1]; + push @long_values, $fields[2]; + for my $alias (@fields[1 .. $#fields]) { + next unless length $alias; + my $loose = loose_name($alias); + die "Joining_Group alias collision for '$alias'\n" + if exists $alias_index{$loose} && $alias_index{$loose} != $index; + $alias_index{$loose} = $index; + die "Joining_Group wildcard alias collision for '$alias'\n" + if exists $wildcard_value_index{$alias} + && $wildcard_value_index{$alias} != $index; + $wildcard_value_index{$alias} = $index; + } +} +die "Expected 106 Joining_Group values, found " . scalar(@short_values) . "\n" + unless @short_values == 106; + +my %property_aliases; +for my $line (split /\n/, $sources[2]{text}) { + next if $line =~ /^\s*#/; + $line =~ s/#.*$//; + my @fields = map { trim($_) } split /;/, $line; + next unless @fields >= 2 && ($fields[0] eq 'jg' || $fields[1] eq 'Joining_Group'); + $property_aliases{loose_name($_)} = 1 for grep { length } @fields; +} +die "Pinned property aliases do not define jg and Joining_Group\n" + unless $property_aliases{jg} && $property_aliases{joininggroup}; + +my (@missing, @explicit); +for my $line (split /\n/, $sources[0]{text}) { + if ($line =~ /^#\s*\@missing:\s*([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z_]+)/) { + my ($range, $value) = ($1, $2); + my ($start, $end) = parse_range($range); + my $index = $alias_index{loose_name($value)}; + die "Unknown \@missing Joining_Group value '$value'\n" unless defined $index; + push @missing, [$start, $end, $index]; + next; + } + next unless $line =~ /^([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z_]+)/; + my ($range, $value) = ($1, $2); + my ($start, $end) = parse_range($range); + my $index = $alias_index{loose_name($value)}; + die "Unknown explicit Joining_Group value '$value'\n" unless defined $index; + push @explicit, [$start, $end, $index]; +} +die "Joining_Group data has no ranges or defaults\n" unless @explicit && @missing; + +my $default_index = $alias_index{loose_name('No_Joining_Group')}; +my @code_value = ($default_index) x 0x110000; +for my $range (@missing) { + my ($code, $end, $index) = @$range; + $code_value[$code++] = $index while $code <= $end; +} +for my $range (@explicit) { + my ($code, $end, $index) = @$range; + $code_value[$code++] = $index while $code <= $end; +} + +my @ranges; +my ($range_start, $range_value) = (0, $code_value[0]); +for my $code (1 .. 0x10ffff) { + next if $code_value[$code] == $range_value; + push @ranges, [$range_start, $code - 1, $range_value]; + ($range_start, $range_value) = ($code, $code_value[$code]); +} +push @ranges, [$range_start, 0x10ffff, $range_value]; + +print <<'HEADER'; +/* + * Generated from Perl 5.44's pinned Unicode Character Database. Do not edit manually. + * +HEADER +for my $source (@sources) { + print " * Source: $source->{name}\n"; + for my $line (split /\n/, $source->{text}) { + next unless $line =~ /^# (?:©|Unicode and|the U\.S\.|For terms of use and license)/; + $line =~ s/^# / * /; + print "$line\n"; + } + print " *\n"; +} +print <<'HEADER_END'; + */ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; + +final class PerlUnicodeJoiningGroupData { +HEADER_END + +print " static final String UNICODE_VERSION = \"$unicode_version\";\n"; +print " static final String DJOIN_GROUP_SHA256 = \"$sources[0]{hash}\";\n"; +print " static final String PROP_VALUE_ALIASES_SHA256 = \"$sources[1]{hash}\";\n"; +print " static final String PROPERTY_ALIASES_SHA256 = \"$sources[2]{hash}\";\n\n"; + +print " private static final String[] SHORT_VALUES = {\n "; +print join(', ', map { qq{"$_"} } @short_values); +print "\n };\n"; +print " private static final String[] LONG_VALUES = {\n "; +print join(', ', map { qq{"$_"} } @long_values); +print "\n };\n"; + +my @aliases = sort keys %alias_index; +print " private static final String[] VALUE_ALIASES = {\n "; +print join(', ', map { qq{"$_"} } @aliases); +print "\n };\n"; +print " private static final byte[] VALUE_ALIAS_INDEX = {\n "; +print join(', ', map { $alias_index{$_} } @aliases); +print "\n };\n"; + +my @wildcard_values = sort keys %wildcard_value_index; +print " private static final String[] WILDCARD_VALUES = {\n "; +print join(', ', map { qq{\"$_\"} } @wildcard_values); +print "\n };\n"; + +my @property_aliases = sort keys %property_aliases; +print " private static final String[] PROPERTY_ALIASES = {\n "; +print join(', ', map { qq{"$_"} } @property_aliases); +print "\n };\n\n"; + +print " private static final int[] RANGES = {\n"; +for (my $i = 0; $i < @ranges; $i += 4) { + my $end = $i + 3 < $#ranges ? $i + 3 : $#ranges; + print " ", join(', ', map { + sprintf '0x%X, 0x%X, %d', @{$ranges[$_]}[0, 1, 2] + } $i .. $end), ",\n"; +} +print <<'FOOTER'; + }; + + private static final UnicodeSet[] VALUE_SETS = buildValueSets(); + + static boolean isPropertyAlias(String alias) { + String loose = looseName(alias); + if (loose == null) return false; + for (String candidate : PROPERTY_ALIASES) { + if (candidate.equals(loose)) return true; + } + return false; + } + + static UnicodeSet valueSet(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : VALUE_SETS[index]; + } + + static String shortValue(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : SHORT_VALUES[index]; + } + + static String canonicalValue(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : LONG_VALUES[index]; + } + + static String[] canonicalValues() { + return LONG_VALUES.clone(); + } + + static String[] wildcardValues() { + return WILDCARD_VALUES.clone(); + } + + private static int valueIndex(String alias) { + String loose = looseName(alias); + if (loose == null) return -1; + for (int i = 0; i < VALUE_ALIASES.length; i++) { + if (VALUE_ALIASES[i].equals(loose)) return VALUE_ALIAS_INDEX[i]; + } + return -1; + } + + private static String looseName(String name) { + if (name == null) return null; + StringBuilder loose = new StringBuilder(name.length()); + for (int i = 0; i < name.length(); i++) { + char character = name.charAt(i); + if (character == '_' || character == '-' || Character.isWhitespace(character)) continue; + loose.append(Character.toLowerCase(character)); + } + return loose.toString(); + } + + private static UnicodeSet[] buildValueSets() { + UnicodeSet[] sets = new UnicodeSet[LONG_VALUES.length]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + for (int i = 0; i < RANGES.length; i += 3) { + sets[RANGES[i + 2]].add(RANGES[i], RANGES[i + 1]); + } + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + + private PerlUnicodeJoiningGroupData() { + } +} +FOOTER diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index 6d0de46109..806762290d 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -379,7 +379,7 @@ my @copy = @{$z}; # ERROR - ✅ **Backreferences to Named Groups**: Using `\k` or `\g{name}` for backreferences to named groups is supported. - ✅ **Relative Backreferences**: Using `\g{-n}` for relative backreferences. - ✅ **Basic Unicode Properties**: Common `\p{...}` and `\P{...}` forms such as `\p{L}` execute through Joni. -- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, and Numeric_Value assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, reserved or composite values, exact rationals, and Perl's generated decimal keyword aliases; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. +- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, Numeric_Value, and Joining_Group assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, reserved or composite values, exact rationals, and Perl's generated decimal keyword aliases; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. - ✅ **Possessive Quantifiers**: Quantifiers like `*+`, `++`, `?+`, and `{n,m}+`, which disable backtracking, are supported. - ✅ **Atomic Grouping**: Use of `(?>...)` for atomic groups is supported. - ✅ **`\K` assertion**: Keep left — in `s///`, text before `\K` is preserved; match variables reflect only the portion after `\K`. @@ -398,7 +398,7 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, Numeric_Value, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 356,848/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. +- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, Numeric_Value, Joining_Group, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 361,138/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. - ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. - ✅ **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter. - ✅ **Unicode Word Boundaries**: Native `\b{wb}`/`\B{wb}` implement WB1–WB16 and WB999 from reproducible Perl 5.44 Unicode 17.0 word-break and extended-pictographic tables. Authoritative chunk 10 passes 19,510/19,510 identically on JVM and interpreter. diff --git a/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeJoiningGroupData.java b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeJoiningGroupData.java new file mode 100644 index 0000000000..b680a8cbf8 --- /dev/null +++ b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeJoiningGroupData.java @@ -0,0 +1,181 @@ +/* + * Generated from Perl 5.44's pinned Unicode Character Database. Do not edit manually. + * + * Source: DerivedJoiningGroup-17.0.0.txt + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + * + * Source: PropertyValueAliases-17.0.0.txt + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + * + * Source: PropertyAliases-17.0.0.txt + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + * + */ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; + +final class PerlUnicodeJoiningGroupData { + static final String UNICODE_VERSION = "17.0.0"; + static final String DJOIN_GROUP_SHA256 = "bb67e0c00b88acfa5be633967b66b23326844a86e49c6fde7b57960d3af66cae"; + static final String PROP_VALUE_ALIASES_SHA256 = "670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01"; + static final String PROPERTY_ALIASES_SHA256 = "4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb"; + + private static final String[] SHORT_VALUES = { + "African_Feh", "African_Noon", "African_Qaf", "Ain", "Alaph", "Alef", "Beh", "Beth", "Burushaski_Yeh_Barree", "Dal", "Dalath_Rish", "E", "Farsi_Yeh", "Fe", "Feh", "Final_Semkath", "Gaf", "Gamal", "Hah", "Hanifi_Rohingya_Kinna_Ya", "Hanifi_Rohingya_Pa", "He", "Heh", "Heh_Goal", "Heth", "Kaf", "Kaph", "Kashmiri_Yeh", "Khaph", "Knotted_Heh", "Lam", "Lamadh", "Malayalam_Bha", "Malayalam_Ja", "Malayalam_Lla", "Malayalam_Llla", "Malayalam_Nga", "Malayalam_Nna", "Malayalam_Nnna", "Malayalam_Nya", "Malayalam_Ra", "Malayalam_Ssa", "Malayalam_Tta", "Manichaean_Aleph", "Manichaean_Ayin", "Manichaean_Beth", "Manichaean_Daleth", "Manichaean_Dhamedh", "Manichaean_Five", "Manichaean_Gimel", "Manichaean_Heth", "Manichaean_Hundred", "Manichaean_Kaph", "Manichaean_Lamedh", "Manichaean_Mem", "Manichaean_Nun", "Manichaean_One", "Manichaean_Pe", "Manichaean_Qoph", "Manichaean_Resh", "Manichaean_Sadhe", "Manichaean_Samekh", "Manichaean_Taw", "Manichaean_Ten", "Manichaean_Teth", "Manichaean_Thamedh", "Manichaean_Twenty", "Manichaean_Waw", "Manichaean_Yodh", "Manichaean_Zayin", "Meem", "Mim", "No_Joining_Group", "Noon", "Nun", "Nya", "Pe", "Qaf", "Qaph", "Reh", "Reversed_Pe", "Rohingya_Yeh", "Sad", "Sadhe", "Seen", "Semkath", "Shin", "Straight_Waw", "Swash_Kaf", "Syriac_Waw", "Tah", "Taw", "Teh_Marbuta", "Teh_Marbuta_Goal", "Teth", "Thin_Noon", "Thin_Yeh", "Vertical_Tail", "Waw", "Yeh", "Yeh_Barree", "Yeh_With_Tail", "Yudh", "Yudh_He", "Zain", "Zhain" + }; + private static final String[] LONG_VALUES = { + "African_Feh", "African_Noon", "African_Qaf", "Ain", "Alaph", "Alef", "Beh", "Beth", "Burushaski_Yeh_Barree", "Dal", "Dalath_Rish", "E", "Farsi_Yeh", "Fe", "Feh", "Final_Semkath", "Gaf", "Gamal", "Hah", "Hanifi_Rohingya_Kinna_Ya", "Hanifi_Rohingya_Pa", "He", "Heh", "Heh_Goal", "Heth", "Kaf", "Kaph", "Kashmiri_Yeh", "Khaph", "Knotted_Heh", "Lam", "Lamadh", "Malayalam_Bha", "Malayalam_Ja", "Malayalam_Lla", "Malayalam_Llla", "Malayalam_Nga", "Malayalam_Nna", "Malayalam_Nnna", "Malayalam_Nya", "Malayalam_Ra", "Malayalam_Ssa", "Malayalam_Tta", "Manichaean_Aleph", "Manichaean_Ayin", "Manichaean_Beth", "Manichaean_Daleth", "Manichaean_Dhamedh", "Manichaean_Five", "Manichaean_Gimel", "Manichaean_Heth", "Manichaean_Hundred", "Manichaean_Kaph", "Manichaean_Lamedh", "Manichaean_Mem", "Manichaean_Nun", "Manichaean_One", "Manichaean_Pe", "Manichaean_Qoph", "Manichaean_Resh", "Manichaean_Sadhe", "Manichaean_Samekh", "Manichaean_Taw", "Manichaean_Ten", "Manichaean_Teth", "Manichaean_Thamedh", "Manichaean_Twenty", "Manichaean_Waw", "Manichaean_Yodh", "Manichaean_Zayin", "Meem", "Mim", "No_Joining_Group", "Noon", "Nun", "Nya", "Pe", "Qaf", "Qaph", "Reh", "Reversed_Pe", "Rohingya_Yeh", "Sad", "Sadhe", "Seen", "Semkath", "Shin", "Straight_Waw", "Swash_Kaf", "Syriac_Waw", "Tah", "Taw", "Teh_Marbuta", "Teh_Marbuta_Goal", "Teth", "Thin_Noon", "Thin_Yeh", "Vertical_Tail", "Waw", "Yeh", "Yeh_Barree", "Yeh_With_Tail", "Yudh", "Yudh_He", "Zain", "Zhain" + }; + private static final String[] VALUE_ALIASES = { + "africanfeh", "africannoon", "africanqaf", "ain", "alaph", "alef", "beh", "beth", "burushaskiyehbarree", "dal", "dalathrish", "e", "farsiyeh", "fe", "feh", "finalsemkath", "gaf", "gamal", "hah", "hamzaonhehgoal", "hanifirohingyakinnaya", "hanifirohingyapa", "he", "heh", "hehgoal", "heth", "kaf", "kaph", "kashmiriyeh", "khaph", "knottedheh", "lam", "lamadh", "malayalambha", "malayalamja", "malayalamlla", "malayalamllla", "malayalamnga", "malayalamnna", "malayalamnnna", "malayalamnya", "malayalamra", "malayalamssa", "malayalamtta", "manichaeanaleph", "manichaeanayin", "manichaeanbeth", "manichaeandaleth", "manichaeandhamedh", "manichaeanfive", "manichaeangimel", "manichaeanheth", "manichaeanhundred", "manichaeankaph", "manichaeanlamedh", "manichaeanmem", "manichaeannun", "manichaeanone", "manichaeanpe", "manichaeanqoph", "manichaeanresh", "manichaeansadhe", "manichaeansamekh", "manichaeantaw", "manichaeanten", "manichaeanteth", "manichaeanthamedh", "manichaeantwenty", "manichaeanwaw", "manichaeanyodh", "manichaeanzayin", "meem", "mim", "nojoininggroup", "noon", "nun", "nya", "pe", "qaf", "qaph", "reh", "reversedpe", "rohingyayeh", "sad", "sadhe", "seen", "semkath", "shin", "straightwaw", "swashkaf", "syriacwaw", "tah", "taw", "tehmarbuta", "tehmarbutagoal", "teth", "thinnoon", "thinyeh", "verticaltail", "waw", "yeh", "yehbarree", "yehwithtail", "yudh", "yudhhe", "zain", "zhain" + }; + private static final byte[] VALUE_ALIAS_INDEX = { + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 93, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105 + }; + private static final String[] WILDCARD_VALUES = { + "African_Feh", "African_Noon", "African_Qaf", "Ain", "Alaph", "Alef", "Beh", "Beth", "Burushaski_Yeh_Barree", "Dal", "Dalath_Rish", "E", "Farsi_Yeh", "Fe", "Feh", "Final_Semkath", "Gaf", "Gamal", "Hah", "Hamza_On_Heh_Goal", "Hanifi_Rohingya_Kinna_Ya", "Hanifi_Rohingya_Pa", "He", "Heh", "Heh_Goal", "Heth", "Kaf", "Kaph", "Kashmiri_Yeh", "Khaph", "Knotted_Heh", "Lam", "Lamadh", "Malayalam_Bha", "Malayalam_Ja", "Malayalam_Lla", "Malayalam_Llla", "Malayalam_Nga", "Malayalam_Nna", "Malayalam_Nnna", "Malayalam_Nya", "Malayalam_Ra", "Malayalam_Ssa", "Malayalam_Tta", "Manichaean_Aleph", "Manichaean_Ayin", "Manichaean_Beth", "Manichaean_Daleth", "Manichaean_Dhamedh", "Manichaean_Five", "Manichaean_Gimel", "Manichaean_Heth", "Manichaean_Hundred", "Manichaean_Kaph", "Manichaean_Lamedh", "Manichaean_Mem", "Manichaean_Nun", "Manichaean_One", "Manichaean_Pe", "Manichaean_Qoph", "Manichaean_Resh", "Manichaean_Sadhe", "Manichaean_Samekh", "Manichaean_Taw", "Manichaean_Ten", "Manichaean_Teth", "Manichaean_Thamedh", "Manichaean_Twenty", "Manichaean_Waw", "Manichaean_Yodh", "Manichaean_Zayin", "Meem", "Mim", "No_Joining_Group", "Noon", "Nun", "Nya", "Pe", "Qaf", "Qaph", "Reh", "Reversed_Pe", "Rohingya_Yeh", "Sad", "Sadhe", "Seen", "Semkath", "Shin", "Straight_Waw", "Swash_Kaf", "Syriac_Waw", "Tah", "Taw", "Teh_Marbuta", "Teh_Marbuta_Goal", "Teth", "Thin_Noon", "Thin_Yeh", "Vertical_Tail", "Waw", "Yeh", "Yeh_Barree", "Yeh_With_Tail", "Yudh", "Yudh_He", "Zain", "Zhain" + }; + private static final String[] PROPERTY_ALIASES = { + "jg", "joininggroup" + }; + + private static final int[] RANGES = { + 0x0, 0x61F, 72, 0x620, 0x620, 27, 0x621, 0x621, 72, 0x622, 0x623, 5, + 0x624, 0x624, 98, 0x625, 0x625, 5, 0x626, 0x626, 99, 0x627, 0x627, 5, + 0x628, 0x628, 6, 0x629, 0x629, 92, 0x62A, 0x62B, 6, 0x62C, 0x62E, 18, + 0x62F, 0x630, 9, 0x631, 0x632, 79, 0x633, 0x634, 84, 0x635, 0x636, 82, + 0x637, 0x638, 90, 0x639, 0x63A, 3, 0x63B, 0x63C, 16, 0x63D, 0x63F, 12, + 0x640, 0x640, 72, 0x641, 0x641, 14, 0x642, 0x642, 77, 0x643, 0x643, 25, + 0x644, 0x644, 30, 0x645, 0x645, 70, 0x646, 0x646, 73, 0x647, 0x647, 22, + 0x648, 0x648, 98, 0x649, 0x64A, 99, 0x64B, 0x66D, 72, 0x66E, 0x66E, 6, + 0x66F, 0x66F, 77, 0x670, 0x670, 72, 0x671, 0x673, 5, 0x674, 0x674, 72, + 0x675, 0x675, 5, 0x676, 0x677, 98, 0x678, 0x678, 99, 0x679, 0x680, 6, + 0x681, 0x687, 18, 0x688, 0x690, 9, 0x691, 0x699, 79, 0x69A, 0x69C, 84, + 0x69D, 0x69E, 82, 0x69F, 0x69F, 90, 0x6A0, 0x6A0, 3, 0x6A1, 0x6A6, 14, + 0x6A7, 0x6A8, 77, 0x6A9, 0x6A9, 16, 0x6AA, 0x6AA, 88, 0x6AB, 0x6AB, 16, + 0x6AC, 0x6AE, 25, 0x6AF, 0x6B4, 16, 0x6B5, 0x6B8, 30, 0x6B9, 0x6BC, 73, + 0x6BD, 0x6BD, 75, 0x6BE, 0x6BE, 29, 0x6BF, 0x6BF, 18, 0x6C0, 0x6C0, 92, + 0x6C1, 0x6C2, 23, 0x6C3, 0x6C3, 93, 0x6C4, 0x6CB, 98, 0x6CC, 0x6CC, 12, + 0x6CD, 0x6CD, 101, 0x6CE, 0x6CE, 12, 0x6CF, 0x6CF, 98, 0x6D0, 0x6D1, 99, + 0x6D2, 0x6D3, 100, 0x6D4, 0x6D4, 72, 0x6D5, 0x6D5, 92, 0x6D6, 0x6ED, 72, + 0x6EE, 0x6EE, 9, 0x6EF, 0x6EF, 79, 0x6F0, 0x6F9, 72, 0x6FA, 0x6FA, 84, + 0x6FB, 0x6FB, 82, 0x6FC, 0x6FC, 3, 0x6FD, 0x6FE, 72, 0x6FF, 0x6FF, 29, + 0x700, 0x70F, 72, 0x710, 0x710, 4, 0x711, 0x711, 72, 0x712, 0x712, 7, + 0x713, 0x714, 17, 0x715, 0x716, 10, 0x717, 0x717, 21, 0x718, 0x718, 89, + 0x719, 0x719, 104, 0x71A, 0x71A, 24, 0x71B, 0x71C, 94, 0x71D, 0x71D, 102, + 0x71E, 0x71E, 103, 0x71F, 0x71F, 26, 0x720, 0x720, 31, 0x721, 0x721, 71, + 0x722, 0x722, 74, 0x723, 0x723, 85, 0x724, 0x724, 15, 0x725, 0x725, 11, + 0x726, 0x726, 76, 0x727, 0x727, 80, 0x728, 0x728, 83, 0x729, 0x729, 78, + 0x72A, 0x72A, 10, 0x72B, 0x72B, 86, 0x72C, 0x72C, 91, 0x72D, 0x72D, 7, + 0x72E, 0x72E, 17, 0x72F, 0x72F, 10, 0x730, 0x74C, 72, 0x74D, 0x74D, 105, + 0x74E, 0x74E, 28, 0x74F, 0x74F, 13, 0x750, 0x756, 6, 0x757, 0x758, 18, + 0x759, 0x75A, 9, 0x75B, 0x75B, 79, 0x75C, 0x75C, 84, 0x75D, 0x75F, 3, + 0x760, 0x761, 14, 0x762, 0x764, 16, 0x765, 0x766, 70, 0x767, 0x769, 73, + 0x76A, 0x76A, 30, 0x76B, 0x76C, 79, 0x76D, 0x76D, 84, 0x76E, 0x76F, 18, + 0x770, 0x770, 84, 0x771, 0x771, 79, 0x772, 0x772, 18, 0x773, 0x774, 5, + 0x775, 0x776, 12, 0x777, 0x777, 99, 0x778, 0x779, 98, 0x77A, 0x77B, 8, + 0x77C, 0x77C, 18, 0x77D, 0x77E, 84, 0x77F, 0x77F, 25, 0x780, 0x85F, 72, + 0x860, 0x860, 36, 0x861, 0x861, 33, 0x862, 0x862, 39, 0x863, 0x863, 42, + 0x864, 0x864, 37, 0x865, 0x865, 38, 0x866, 0x866, 32, 0x867, 0x867, 40, + 0x868, 0x868, 34, 0x869, 0x869, 35, 0x86A, 0x86A, 41, 0x86B, 0x86F, 72, + 0x870, 0x882, 5, 0x883, 0x885, 72, 0x886, 0x886, 96, 0x887, 0x888, 72, + 0x889, 0x889, 73, 0x88A, 0x88A, 18, 0x88B, 0x88C, 90, 0x88D, 0x88D, 16, + 0x88E, 0x88E, 97, 0x88F, 0x88F, 73, 0x890, 0x89F, 72, 0x8A0, 0x8A1, 6, + 0x8A2, 0x8A2, 18, 0x8A3, 0x8A3, 90, 0x8A4, 0x8A4, 14, 0x8A5, 0x8A5, 77, + 0x8A6, 0x8A6, 30, 0x8A7, 0x8A7, 70, 0x8A8, 0x8A9, 99, 0x8AA, 0x8AA, 79, + 0x8AB, 0x8AB, 98, 0x8AC, 0x8AC, 81, 0x8AD, 0x8AD, 72, 0x8AE, 0x8AE, 9, + 0x8AF, 0x8AF, 82, 0x8B0, 0x8B0, 16, 0x8B1, 0x8B1, 87, 0x8B2, 0x8B2, 79, + 0x8B3, 0x8B3, 3, 0x8B4, 0x8B4, 25, 0x8B5, 0x8B5, 77, 0x8B6, 0x8B8, 6, + 0x8B9, 0x8B9, 79, 0x8BA, 0x8BA, 99, 0x8BB, 0x8BB, 0, 0x8BC, 0x8BC, 2, + 0x8BD, 0x8BD, 1, 0x8BE, 0x8C0, 6, 0x8C1, 0x8C1, 18, 0x8C2, 0x8C2, 16, + 0x8C3, 0x8C3, 3, 0x8C4, 0x8C4, 2, 0x8C5, 0x8C6, 18, 0x8C7, 0x8C7, 30, + 0x8C8, 0x8C8, 16, 0x8C9, 0x10ABF, 72, 0x10AC0, 0x10AC0, 43, 0x10AC1, 0x10AC2, 45, + 0x10AC3, 0x10AC4, 49, 0x10AC5, 0x10AC5, 46, 0x10AC6, 0x10AC6, 72, 0x10AC7, 0x10AC7, 67, + 0x10AC8, 0x10AC8, 72, 0x10AC9, 0x10ACA, 69, 0x10ACB, 0x10ACC, 72, 0x10ACD, 0x10ACD, 50, + 0x10ACE, 0x10ACE, 64, 0x10ACF, 0x10ACF, 68, 0x10AD0, 0x10AD2, 52, 0x10AD3, 0x10AD3, 53, + 0x10AD4, 0x10AD4, 47, 0x10AD5, 0x10AD5, 65, 0x10AD6, 0x10AD6, 54, 0x10AD7, 0x10AD7, 55, + 0x10AD8, 0x10AD8, 61, 0x10AD9, 0x10ADA, 44, 0x10ADB, 0x10ADC, 57, 0x10ADD, 0x10ADD, 60, + 0x10ADE, 0x10AE0, 58, 0x10AE1, 0x10AE1, 59, 0x10AE2, 0x10AE3, 72, 0x10AE4, 0x10AE4, 62, + 0x10AE5, 0x10AEA, 72, 0x10AEB, 0x10AEB, 56, 0x10AEC, 0x10AEC, 48, 0x10AED, 0x10AED, 63, + 0x10AEE, 0x10AEE, 66, 0x10AEF, 0x10AEF, 51, 0x10AF0, 0x10D01, 72, 0x10D02, 0x10D02, 20, + 0x10D03, 0x10D08, 72, 0x10D09, 0x10D09, 20, 0x10D0A, 0x10D18, 72, 0x10D19, 0x10D19, 19, + 0x10D1A, 0x10D1B, 72, 0x10D1C, 0x10D1C, 20, 0x10D1D, 0x10D1D, 72, 0x10D1E, 0x10D1E, 19, + 0x10D1F, 0x10D1F, 72, 0x10D20, 0x10D20, 19, 0x10D21, 0x10D22, 72, 0x10D23, 0x10D23, 19, + 0x10D24, 0x10EC1, 72, 0x10EC2, 0x10EC2, 9, 0x10EC3, 0x10EC3, 90, 0x10EC4, 0x10EC4, 25, + 0x10EC5, 0x10EC5, 72, 0x10EC6, 0x10EC6, 95, 0x10EC7, 0x10EC7, 99, 0x10EC8, 0x10FFFF, 72, + }; + + private static final UnicodeSet[] VALUE_SETS = buildValueSets(); + + static boolean isPropertyAlias(String alias) { + String loose = looseName(alias); + if (loose == null) return false; + for (String candidate : PROPERTY_ALIASES) { + if (candidate.equals(loose)) return true; + } + return false; + } + + static UnicodeSet valueSet(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : VALUE_SETS[index]; + } + + static String shortValue(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : SHORT_VALUES[index]; + } + + static String canonicalValue(String alias) { + int index = valueIndex(alias); + return index < 0 ? null : LONG_VALUES[index]; + } + + static String[] canonicalValues() { + return LONG_VALUES.clone(); + } + + static String[] wildcardValues() { + return WILDCARD_VALUES.clone(); + } + + private static int valueIndex(String alias) { + String loose = looseName(alias); + if (loose == null) return -1; + for (int i = 0; i < VALUE_ALIASES.length; i++) { + if (VALUE_ALIASES[i].equals(loose)) return VALUE_ALIAS_INDEX[i]; + } + return -1; + } + + private static String looseName(String name) { + if (name == null) return null; + StringBuilder loose = new StringBuilder(name.length()); + for (int i = 0; i < name.length(); i++) { + char character = name.charAt(i); + if (character == '_' || character == '-' || Character.isWhitespace(character)) continue; + loose.append(Character.toLowerCase(character)); + } + return loose.toString(); + } + + private static UnicodeSet[] buildValueSets() { + UnicodeSet[] sets = new UnicodeSet[LONG_VALUES.length]; + for (int i = 0; i < sets.length; i++) sets[i] = new UnicodeSet(); + for (int i = 0; i < RANGES.length; i += 3) { + sets[RANGES[i + 2]].add(RANGES[i], RANGES[i + 1]); + } + for (UnicodeSet set : sets) set.freeze(); + return sets; + } + + private PerlUnicodeJoiningGroupData() { + } +} diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index 32c9072324..a74ccda49a 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -699,11 +699,17 @@ private static String translateUnicodeProperty(String property, boolean negated, } boolean isPrefixedNumericWildcard = isPerlIsPrefixedNumericWildcard(property); + boolean isPrefixedJoiningGroupWildcard = + isPerlIsPrefixedJoiningGroupWildcard(property); property = normalizePerlIsPropertyAssignment(property); if (isPrefixedNumericWildcard) { throw new IllegalArgumentException( "Is-prefixed Numeric_Value properties do not accept wildcard values"); } + if (isPrefixedJoiningGroupWildcard) { + throw new IllegalArgumentException( + "Can't find Unicode property definition for Is-prefixed Joining_Group wildcard"); + } if (property.startsWith("utf8::")) { String userPropertyName = property.substring("utf8::".length()); if (!userPropertyName.matches("[A-Za-z_][A-Za-z0-9_]*")) { @@ -926,6 +932,7 @@ private static String translateUnicodeProperty(String property, boolean negated, String message = e.getMessage(); if (message != null && (message.contains("in expansion of") || message.startsWith("Illegal user-defined property name") + || message.startsWith("Can't find Unicode property definition") || message.startsWith("Timeout waiting for another thread"))) { throw e; } @@ -1017,6 +1024,11 @@ && isCanonicalCombiningClassProperty(alias.substring(0, assignment))) { } return numericValue; } + if (assignment > 0 && assignment < alias.length() - 1 + && PerlUnicodeJoiningGroupData.isPropertyAlias( + alias.substring(0, assignment))) { + return resolvePerlJoiningGroup(alias.substring(assignment + 1)); + } if (assignment > 0 && assignment < alias.length() - 1) { Boolean value = perlBooleanPropertyValue(alias.substring(assignment + 1)); if (value != null) { @@ -1080,13 +1092,75 @@ private static boolean isPerlIsPrefixedNumericWildcard(String property) { int assignment = propertyValueDelimiter(property); if (assignment <= 0 || assignment == property.length() - 1) return false; String name = property.substring(0, assignment); - return name.length() > 2 - && name.charAt(0) == 'I' - && name.charAt(1) == 's' - && PerlUnicodeNumericValueData.isPropertyAlias(name) + String propertyName = exactIsPrefixedPropertyName(name); + return propertyName != null + && PerlUnicodeNumericValueData.isPropertyAlias(propertyName) && perlNumericWildcardBody(property.substring(assignment + 1)) != null; } + private static boolean isPerlIsPrefixedJoiningGroupWildcard(String property) { + int assignment = propertyValueDelimiter(property); + if (assignment <= 0 || assignment == property.length() - 1) return false; + String name = property.substring(0, assignment); + String propertyName = exactIsPrefixedPropertyName(name); + return propertyName != null + && PerlUnicodeJoiningGroupData.isPropertyAlias(propertyName) + && perlNumericWildcardBody(property.substring(assignment + 1)) != null; + } + + private static String exactIsPrefixedPropertyName(String name) { + if (name.length() <= 2 || name.charAt(0) != 'I' || name.charAt(1) != 's') { + return null; + } + int start = 2; + while (start < name.length()) { + char separator = name.charAt(start); + if (!Character.isWhitespace(separator) + && separator != '-' && separator != '_') { + break; + } + start++; + } + return start == name.length() ? null : name.substring(start); + } + + private static UnicodeSet resolvePerlJoiningGroup(String value) { + String wildcard = perlNumericWildcardBody(value); + if (wildcard == null) { + UnicodeSet exact = PerlUnicodeJoiningGroupData.valueSet(value); + if (exact == null) { + throw new IllegalArgumentException( + "Unsupported Joining_Group value: " + value.trim()); + } + return exact; + } + if (wildcard.indexOf('*') >= 0) { + throw new IllegalArgumentException( + "quantifier '*' is not allowed in Unicode property value wildcard"); + } + + Pattern valuePattern; + try { + valuePattern = Pattern.compile(wildcard); + } catch (RuntimeException invalidPattern) { + throw new IllegalArgumentException( + "Invalid Unicode property value wildcard", invalidPattern); + } + + UnicodeSet result = new UnicodeSet(); + for (String candidate : PerlUnicodeJoiningGroupData.wildcardValues()) { + if (valuePattern.matcher(candidate).matches() + || valuePattern.matcher(loosePropertyName(candidate)).matches()) { + result.addAll(PerlUnicodeJoiningGroupData.valueSet(candidate)); + } + } + if (result.isEmpty()) { + throw new IllegalArgumentException( + "No Unicode property value wildcard matches Joining_Group"); + } + return result.freeze(); + } + private static UnicodeSet resolvePerlNumericValue(String value) { String wildcard = perlNumericWildcardBody(value); if (wildcard != null) { diff --git a/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeJoiningGroupDataTest.java b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeJoiningGroupDataTest.java new file mode 100644 index 0000000000..323093084f --- /dev/null +++ b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeJoiningGroupDataTest.java @@ -0,0 +1,102 @@ +package org.perlonjava.runtime.regex; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.Arrays; + +import com.ibm.icu.text.UnicodeSet; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; + +@Tag("unit") +class PerlUnicodeJoiningGroupDataTest { + @Test + void usesChecksumPinnedPerl544Unicode17Sources() { + assertEquals("17.0.0", PerlUnicodeJoiningGroupData.UNICODE_VERSION); + assertEquals("bb67e0c00b88acfa5be633967b66b23326844a86e49c6fde7b57960d3af66cae", + PerlUnicodeJoiningGroupData.DJOIN_GROUP_SHA256); + assertEquals("670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01", + PerlUnicodeJoiningGroupData.PROP_VALUE_ALIASES_SHA256); + assertEquals("4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb", + PerlUnicodeJoiningGroupData.PROPERTY_ALIASES_SHA256); + } + + @Test + void acceptsPinnedPropertyAndLooseValueAliases() { + assertTrue(PerlUnicodeJoiningGroupData.isPropertyAlias("jg")); + assertTrue(PerlUnicodeJoiningGroupData.isPropertyAlias("Joining Group")); + assertTrue(PerlUnicodeJoiningGroupData.isPropertyAlias("JOINING-GROUP")); + assertFalse(PerlUnicodeJoiningGroupData.isPropertyAlias("Joining Type")); + + UnicodeSet africanFeh = PerlUnicodeJoiningGroupData.valueSet("African_Feh"); + assertSame(africanFeh, + PerlUnicodeJoiningGroupData.valueSet("a-f r_i c a n feh")); + assertEquals("African_Feh", + PerlUnicodeJoiningGroupData.shortValue("african feh")); + assertEquals("African_Feh", + PerlUnicodeJoiningGroupData.canonicalValue("AFRICAN-FEH")); + assertNull(PerlUnicodeJoiningGroupData.valueSet("not a joining group")); + assertNull(PerlUnicodeJoiningGroupData.canonicalValue(null)); + } + + @Test + void retainsEveryPinnedValueAndWildcardAlias() { + String[] values = PerlUnicodeJoiningGroupData.canonicalValues(); + assertEquals(106, values.length); + for (String value : values) { + UnicodeSet set = PerlUnicodeJoiningGroupData.valueSet(value); + assertNotNull(set, value); + assertTrue(set.isFrozen(), value); + assertFalse(set.isEmpty(), value); + assertEquals(value, PerlUnicodeJoiningGroupData.shortValue(value), value); + } + + String[] wildcardValues = PerlUnicodeJoiningGroupData.wildcardValues(); + assertEquals(107, wildcardValues.length); + for (String value : wildcardValues) { + assertNotNull(PerlUnicodeJoiningGroupData.valueSet(value), value); + } + assertTrue(Arrays.asList(wildcardValues).contains("Hamza_On_Heh_Goal")); + } + + @Test + void mapsTheAlternateTehMarbutaGoalAlias() { + UnicodeSet canonical = PerlUnicodeJoiningGroupData.valueSet("Teh_Marbuta_Goal"); + assertSame(canonical, PerlUnicodeJoiningGroupData.valueSet("Hamza_On_Heh_Goal")); + assertEquals("Teh_Marbuta_Goal", + PerlUnicodeJoiningGroupData.canonicalValue("hamza on heh goal")); + assertTrue(canonical.contains(0x06C3)); + } + + @Test + void appliesNoJoiningGroupDefaultBeforeExplicitRanges() { + UnicodeSet none = PerlUnicodeJoiningGroupData.valueSet("No_Joining_Group"); + assertTrue(none.contains(0x0000)); + assertTrue(none.contains(0x0378)); + assertTrue(none.contains(0x10FFFF)); + assertFalse(none.contains(0x0639)); + assertTrue(PerlUnicodeJoiningGroupData.valueSet("Ain").contains(0x0639)); + } + + @Test + void formsOneDisjointPartitionOfAllUnicodeCodePoints() { + String[] values = PerlUnicodeJoiningGroupData.canonicalValues(); + UnicodeSet union = new UnicodeSet(); + for (int i = 0; i < values.length; i++) { + UnicodeSet current = PerlUnicodeJoiningGroupData.valueSet(values[i]); + union.addAll(current); + for (int j = i + 1; j < values.length; j++) { + UnicodeSet overlap = new UnicodeSet(current) + .retainAll(PerlUnicodeJoiningGroupData.valueSet(values[j])); + assertTrue(overlap.isEmpty(), values[i] + " overlaps " + values[j]); + } + } + assertEquals(0x110000, union.size()); + assertTrue(union.contains(0, 0x10FFFF)); + } +} diff --git a/src/test/resources/unit/regex/unicode_joining_group_pinned.t b/src/test/resources/unit/regex/unicode_joining_group_pinned.t new file mode 100644 index 0000000000..ccc5fd1e63 --- /dev/null +++ b/src/test/resources/unit/regex/unicode_joining_group_pinned.t @@ -0,0 +1,92 @@ +use strict; +use warnings; +use Test::More; +no warnings 'experimental::uniprop_wildcards'; + +my $joining_group_probe = eval q{qr/\p{jg=No_Joining_Group}/}; +if (!defined $joining_group_probe) { + plan skip_all => 'pinned Joining_Group data awaits resolver integration'; +} + +my @values = ( + ['African_Feh', 'African_Feh', 0x08BB], + ['Ain', 'Ain', 0x0639], + ['Alef', 'Alef', 0x0622], + ['Manichaean_Aleph', 'Manichaean_Aleph', 0x10AC0], + ['No_Joining_Group', 'No_Joining_Group', 0x0000], + ['Teh_Marbuta_Goal', 'Teh_Marbuta_Goal', 0x06C3], +); + +for my $value (@values) { + my ($short, $long, $code_point) = @$value; + my $character = chr($code_point); + my $short_pattern = eval "qr/\\p{jg=$short}/"; + ok(defined $short_pattern, "short value $short compiles") + or diag($@); + like($character, $short_pattern, "short value $short matches representative"); + + my $long_pattern = eval "qr/\\p{Joining_Group=$long}/"; + ok(defined $long_pattern, "long value $long compiles") + or diag($@); + like($character, $long_pattern, "long value $long matches representative"); +} + +SKIP: { + my $short_pattern = eval q{qr/\p{jg=Thin_Noon}/}; + skip 'Thin_Noon requires Perl 5.44 Unicode 17 data', 4 + unless defined $short_pattern; + ok(defined $short_pattern, 'Unicode 17 short value Thin_Noon compiles'); + like(chr(0x10EC6), $short_pattern, + 'Unicode 17 short value Thin_Noon matches representative'); + my $long_pattern = eval q{qr/\p{Joining_Group=Thin_Noon}/}; + ok(defined $long_pattern, 'Unicode 17 long value Thin_Noon compiles'); + like(chr(0x10EC6), $long_pattern, + 'Unicode 17 long value Thin_Noon matches representative'); +} + +my $loose = eval q{qr/\p{joining group = african feh}/}; +ok(defined $loose, 'loose property and value aliases compile') or diag($@); +like(chr(0x08BB), $loose, 'loose property and value aliases match'); + +my $alternate = eval q{qr/\p{jg=Hamza_On_Heh_Goal}/}; +ok(defined $alternate, 'alternate value alias compiles') or diag($@); +like(chr(0x06C3), $alternate, 'alternate value alias matches canonical set'); + +my $no_joining_group = eval q{qr/\p{jg=No_Joining_Group}/}; +like(chr(0x0378), $no_joining_group, + 'unassigned code point uses No_Joining_Group default'); +like(chr(0x10FFFF), $no_joining_group, + 'final code point uses No_Joining_Group default'); +unlike(chr(0x0639), $no_joining_group, + 'explicit Ain range overrides No_Joining_Group default'); + +my @accepted_wildcards = ( + [q{qr/\p{jg=:\AAfrican_.+\z:}/}, 0x08BB, 'canonical wildcard'], + [q{qr/\p{jg=:\Aafrican.+\z:}/}, 0x08BB, 'squeezed wildcard'], + [q{qr/\p{jg=:\A(?:Ain|Alef)\z:}/}, 0x0639, 'selected wildcard'], + [q{qr/\p{jg=:\AHamza_On_Heh_Goal\z:}/}, 0x06C3, + 'alternate-alias wildcard'], +); +for my $wildcard (@accepted_wildcards) { + my ($source, $code_point, $description) = @$wildcard; + my $pattern = eval $source; + ok(defined $pattern, "$description compiles") or diag($@); + like(chr($code_point), $pattern, "$description matches"); +} + +my @rejected_wildcards = ( + [q{qr/\p{jg=:\ANot_A_Value\z:}/}, + qr/No Unicode property value wildcard matches/, 'unmatched wildcard'], + [q{qr/\p{Is_jg=:\AAin\z:}/}, + qr/Can't find Unicode property definition/, 'Is-prefixed wildcard'], + [q{qr/\p{jg=:\A.*\z:}/}, + qr/quantifier '\*' is not allowed/i, 'star-quantifier wildcard'], +); +for my $wildcard (@rejected_wildcards) { + my ($source, $error_pattern, $description) = @$wildcard; + my $pattern = eval $source; + ok(!defined $pattern, "$description is rejected"); + like($@, $error_pattern, "$description reports the expected diagnostic"); +} + +done_testing(); From b0cdc8d6f3f047ae7a5ea33bfd377f7434cc3bf4 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:14:54 +0200 Subject: [PATCH 34/60] feat(regex): add pinned Block parity (#1039) Squashed integration of PR #1039 at 5d5f6909d. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 33 +- dev/tools/generate_perl_unicode_block_data.pl | 336 +++++++++++++++ docs/reference/feature-matrix.md | 4 +- .../runtime/regex/PerlUnicodeBlockData.java | 399 ++++++++++++++++++ .../runtime/regex/UnicodeResolver.java | 131 +++++- .../regex/PerlUnicodeBlockDataTest.java | 158 +++++++ .../resources/unit/regex/unicode_block_data.t | 84 ++++ 7 files changed, 1125 insertions(+), 20 deletions(-) create mode 100644 dev/tools/generate_perl_unicode_block_data.pl create mode 100644 src/main/java/org/perlonjava/runtime/regex/PerlUnicodeBlockData.java create mode 100644 src/test/java/org/perlonjava/runtime/regex/PerlUnicodeBlockDataTest.java create mode 100644 src/test/resources/unit/regex/unicode_block_data.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 3df6dcabb3..330dfa6d9b 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,7 +224,7 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1037, stacked +The published integration stack is preserved through draft PR #1038, stacked on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, lossless generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the @@ -238,10 +238,11 @@ line boundaries. PR #1031 adds pinned Canonical_Combining_Class sets and valid empty-property rendering. PR #1032 integrates native numeric escapes through U+10FFFF; PR #1033 adds pinned Bidi_Class sets; PR #1034 integrates native vertical-whitespace escapes; PR #1035 adds Decomposition_Type and PR #1036 -adds East_Asian_Width and PR #1037 adds Numeric_Value. The current WIP -integrates Joining_Group; independently validated Block, -Script/Script_Extensions, break-property values, and generic and specialized -binary-property data are ready for focused resolver integration. +adds East_Asian_Width, PR #1037 adds Numeric_Value, and PR #1038 adds +Joining_Group. The current WIP integrates Block; independently validated +Script/Script_Extensions, break-property values, generic and specialized +binary-property data, and the first preprocessor dead-state deletion are ready +for focused integration. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -371,6 +372,18 @@ Chunks 01–04 gain 4,290 assertions with no pass-count regression and exact JVM/interpreter success sets, raising current generated evidence to 361,138/407,367. +`Block`/`blk` assignments and `In...`/single-`Is...` shortcuts now resolve all +347 values, including `No_Block`, from a complete pinned Unicode 17 partition. +Official compact aliases, loose forms, `#...#` wildcards, Script and +General_Category/binary precedence, ordered gaps, noncharacters, and exact +compound `Is` policy follow Perl 5.44. The 36-assertion oracle passes standard +Perl; JVM and interpreter pass all 35 Block-specific assertions while retaining +one pre-existing TODO for unresolved deferred `In...` user-property timing. +The two focused precedence reducers pass 12/12 on all runtimes, protected +boundary smoke remains 169/169 per backend, and chunks 01–04 gain 8,324 +assertions with zero numbered regressions and exact backend identity. Current +generated evidence is 369,462/407,367. + Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while ordinary and substitution callbacks execute, without misclassifying escaped or @@ -640,6 +653,10 @@ is retained for now. partition with loose aliases, ordered defaults, alternate wildcard names, and exact `Is`/wildcard rejection policy. The focused oracle passes 49/49; chunks 01–04 gain 4,290 assertions with exact backend identity. + - [x] Generated and integrated a complete pinned Unicode 17.0 Block + partition with official aliases, ordered `No_Block` gaps, wildcard policy, + and Script/category/binary precedence. Chunks 01–04 gain 8,324 assertions + with zero numbered regressions and exact backend identity. - [x] Integrated native Perl `\v`/`\V` dispatch inside and outside character classes (`1eff1db97`, integrated as `6328935cd`). The focused oracle passes 92/92 and unchanged `reg_posixcc.t` passes 2,560/2,560 on both backends. @@ -685,13 +702,13 @@ is retained for now. ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1037. Publish the validated - Joining_Group integration in a separate focused WIP PR, preserving the +1. Land review-ready PR #1024 and draft PRs #1025–#1038. Publish the validated + Block integration in a separate focused WIP PR, preserving the independently reviewed data commit. 2. Preserve the now-complete native Joni boundary corpus at 239,866/239,866: sentence chunk 05, line chunks 06–09, and word chunk 10 must remain exact on JVM and interpreter while property and parser work continues. -3. Integrate the independently generated Block, Script/Script_Extensions, and +3. Integrate the independently generated Script/Script_Extensions and break-property value slices, then the generic and specialized binary-property families currently advancing in parallel. Preserve pinned Perl 5.44 diff --git a/dev/tools/generate_perl_unicode_block_data.pl b/dev/tools/generate_perl_unicode_block_data.pl new file mode 100644 index 0000000000..b7cec0cec2 --- /dev/null +++ b/dev/tools/generate_perl_unicode_block_data.pl @@ -0,0 +1,336 @@ +#!/usr/bin/env perl +use strict; +use warnings; +use Digest::SHA qw(sha256_hex); +use File::Spec; +use FindBin; + +binmode STDOUT, ':raw'; + +my $expected_version = '17.0.0'; +my $unicore = File::Spec->catdir($FindBin::Bin, '..', '..', 'perl5', 'lib', 'unicore'); +my %sources = ( + Version => [File::Spec->catfile($unicore, 'version'), + '8c30575264b2772c7a69c5bb6069a28f0e0a7a0df735871bde2d99ee674316ac'], + Blocks => [File::Spec->catfile($unicore, 'Blocks.txt'), + 'c0edefaf1a19771e830a82735472716af6bf3c3975f6c2a23ffbe2580fbbcb15'], + Property_Aliases => [File::Spec->catfile($unicore, 'PropertyAliases.txt'), + '4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb'], + Property_Value_Aliases => [File::Spec->catfile($unicore, 'PropValueAliases.txt'), + '670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01'], +); + +sub source_text { + my ($name) = @_; + my ($path, $expected_hash) = @{$sources{$name}}; + open my $input, '<:raw', $path or die "Cannot read $path: $!\n"; + local $/; + my $text = <$input>; + close $input or die "Cannot close $path: $!\n"; + my $actual_hash = sha256_hex($text); + die "$path SHA-256 mismatch: expected $expected_hash, found $actual_hash\n" + unless $actual_hash eq $expected_hash; + return ($path, $text); +} + +sub trim { + my ($text) = @_; + $text =~ s/^\s+|\s+$//g; + return $text; +} + +sub loose { + my ($text) = @_; + $text = lc $text; + $text =~ s/[\x09-\x0d _-]+//g; + return $text; +} + +sub range_from_text { + my ($range) = @_; + my ($first, $last) = split /\.\./, $range; + return (hex($first), hex(defined $last ? $last : $first)); +} + +sub verify_unicode_notice { + my ($path, $text) = @_; + die "$path does not preserve the Unicode copyright notice\n" + unless $text =~ /^# © 2025 Unicode®, Inc\.$/m; + die "$path does not preserve the Unicode trademark notice\n" + unless $text =~ /^# Unicode and the Unicode Logo are registered trademarks of Unicode, Inc\. in the U\.S\. and other countries\.$/m; + die "$path does not preserve the Unicode terms notice\n" + unless $text =~ m{^# For terms of use and license, see https://www\.unicode\.org/terms_of_use\.html$}m; +} + +my ($version_path, $version_text) = source_text('Version'); +$version_text =~ s/\s+\z//; +die "Expected Unicode $expected_version, found '$version_text' in $version_path\n" + unless $version_text eq $expected_version; + +my ($property_path, $property_text) = source_text('Property_Aliases'); +die "$property_path is not pinned Unicode $expected_version data\n" + unless $property_text =~ /^# PropertyAliases-\Q$expected_version\E\.txt$/m; +verify_unicode_notice($property_path, $property_text); +my @property_aliases; +for my $line (split /\n/, $property_text) { + $line =~ s/#.*//; + my @fields = map { trim($_) } split /;/, $line, -1; + next unless @fields >= 2 && $fields[0] eq 'blk'; + @property_aliases = grep { length } @fields; +} +die "Unexpected Block aliases in $property_path: @property_aliases\n" + unless join("\0", @property_aliases) eq join("\0", 'blk', 'Block'); + +my ($value_path, $value_text) = source_text('Property_Value_Aliases'); +die "$value_path is not pinned Unicode $expected_version data\n" + unless $value_text =~ /^# PropertyValueAliases-\Q$expected_version\E\.txt$/m; +verify_unicode_notice($value_path, $value_text); +my (@value_rows, %row_for_long, %row_for_alias); +my $in_block_values = 0; +for my $line (split /\n/, $value_text) { + if ($line =~ /^# Block \(blk\)$/) { + $in_block_values = 1; + next; + } + last if $in_block_values && $line =~ /^# /; + next unless $in_block_values && $line =~ /^blk\s*;/; + $line =~ s/#.*//; + my @fields = map { trim($_) } split /;/, $line, -1; + shift @fields; + @fields = grep { length } @fields; + die "Block value record lacks short/long aliases: '$line'\n" unless @fields >= 2; + my $row = scalar @value_rows; + push @value_rows, \@fields; + my $long_key = loose($fields[1]); + die "Duplicate Block long alias '$fields[1]'\n" if exists $row_for_long{$long_key}; + $row_for_long{$long_key} = $row; + for my $alias (@fields) { + my $key = loose($alias); + die "Block alias '$alias' collides across values\n" + if exists $row_for_alias{$key} && $row_for_alias{$key} != $row; + $row_for_alias{$key} = $row; + } +} +die "Expected 347 Block value records, found " . scalar(@value_rows) . "\n" + unless @value_rows == 347; +my %exact_aliases; +my $alias_field_count = 0; +for my $row (@value_rows) { + $alias_field_count += @$row; + $exact_aliases{$_} = 1 for @$row; +} +die "Expected 700 Block alias fields, found $alias_field_count\n" + unless $alias_field_count == 700; +die "Expected 496 exact Block aliases, found " . scalar(keys %exact_aliases) . "\n" + unless keys(%exact_aliases) == 496; +die "Expected 495 loose Block aliases, found " . scalar(keys %row_for_alias) . "\n" + unless keys(%row_for_alias) == 495; +my $no_block_row = $row_for_long{loose('No_Block')}; +die "Missing No_Block aliases\n" unless defined $no_block_row; + +my ($blocks_path, $blocks_text) = source_text('Blocks'); +die "$blocks_path is not pinned Unicode $expected_version data\n" + unless $blocks_text =~ /^# Blocks-\Q$expected_version\E\.txt$/m; +verify_unicode_notice($blocks_path, $blocks_text); +my (@missing, @explicit); +for my $line (split /\n/, $blocks_text) { + if ($line =~ /^\#\s*\@missing:\s*([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*; + \s*([^#]+?)\s*$/x) { + my ($first, $last) = range_from_text($1); + push @missing, [$first, $last, trim($2)]; + next; + } + next if $line =~ /^\s*(?:#|$)/; + die "Malformed Blocks record '$line'\n" + unless $line =~ /^([0-9A-F]+\.\.[0-9A-F]+)\s*;\s*([^#]+?)\s*$/; + my ($first, $last) = range_from_text($1); + push @explicit, [$first, $last, trim($2)]; +} +die "Expected one ordered Block \@missing rule\n" + unless @missing == 1 && $missing[0][0] == 0 && $missing[0][1] == 0x10ffff + && loose($missing[0][2]) eq loose('No_Block'); +die "Expected 346 explicit Block ranges, found " . scalar(@explicit) . "\n" + unless @explicit == 346; + +my @value_names = ('No_Block'); +my %value_id_for_row = ($no_block_row => 0); +my $explicit_count = 0; +for my $index (0 .. $#explicit) { + my ($first, $last, $source_name) = @{$explicit[$index]}; + die sprintf("Invalid Block range U+%04X..U+%04X\n", $first, $last) + if $first < 0 || $last > 0x10ffff || $first > $last; + die sprintf("Block range is not aligned to hex columns: U+%04X..U+%04X\n", $first, $last) + if ($first & 0xf) != 0 || ($last & 0xf) != 0xf; + die sprintf("Overlapping or out-of-order Block range at U+%04X\n", $first) + if $index > 0 && $first <= $explicit[$index - 1][1]; + my $row = $row_for_long{loose($source_name)}; + die "Block '$source_name' has no long value alias\n" unless defined $row; + die "Block '$source_name' is represented by more than one range\n" + if exists $value_id_for_row{$row}; + my $value_id = scalar @value_names; + $value_id_for_row{$row} = $value_id; + push @value_names, $value_rows[$row][1]; + $explicit[$index][2] = $value_id; + $explicit_count += $last - $first + 1; +} +die "Expected 347 reachable Block values, found " . scalar(@value_names) . "\n" + unless @value_names == 347 && keys(%value_id_for_row) == 347; +die "Expected 303,808 named-Block code points, found $explicit_count\n" + unless $explicit_count == 303_808; + +my @partition; +my $cursor = 0; +for my $range (@explicit) { + push @partition, [$cursor, $range->[0] - 1, 0] if $cursor < $range->[0]; + push @partition, [@$range]; + $cursor = $range->[1] + 1; +} +push @partition, [$cursor, 0x10ffff, 0] if $cursor <= 0x10ffff; +die "Expected 397 complete Block intervals, found " . scalar(@partition) . "\n" + unless @partition == 397; +my ($partition_count, $no_block_count, $no_block_ranges) = (0, 0, 0); +for my $index (0 .. $#partition) { + my ($first, $last, $value_id) = @{$partition[$index]}; + die "Block partition is not contiguous\n" + if ($index == 0 && $first != 0) + || ($index > 0 && $first != $partition[$index - 1][1] + 1); + die "Block partition value is out of bounds\n" + if $value_id < 0 || $value_id >= @value_names; + my $count = $last - $first + 1; + $partition_count += $count; + if ($value_id == 0) { + $no_block_count += $count; + $no_block_ranges++; + } +} +die "Block partition does not end at U+10FFFF\n" + unless $partition[-1][1] == 0x10ffff; +die "Block partition does not cover the Unicode scalar universe\n" + unless $partition_count == 0x110000; +die "Expected 51 No_Block ranges and 810,304 code points\n" + unless $no_block_ranges == 51 && $no_block_count == 810_304; + +my %alias_value_id; +for my $key (keys %row_for_alias) { + my $value_id = $value_id_for_row{$row_for_alias{$key}}; + die "Block alias '$key' has no reachable value\n" unless defined $value_id; + $alias_value_id{$key} = $value_id; +} +my @alias_keys = sort keys %alias_value_id; + +print <<'HEADER'; +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; +import java.util.Arrays; + +/* + * Generated from Perl 5.44's pinned Unicode Character Database by + * dev/tools/generate_perl_unicode_block_data.pl. Do not edit manually. + * + * Unicode data source copyright: + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in + * the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + */ +final class PerlUnicodeBlockData { +HEADER + +print " static final String UNICODE_VERSION = \"$expected_version\";\n"; +print " static final short INVALID = -1;\n"; +print " static final short NO_BLOCK = 0;\n\n"; +print " private static final String[] VALUE_NAMES = {\n"; +for (my $i = 0; $i < @value_names; $i += 6) { + my $end = $i + 5 < $#value_names ? $i + 5 : $#value_names; + print " ", join(', ', map { qq{\"$value_names[$_]\"} } $i .. $end), ",\n"; +} +print " };\n\n private static final int[] RANGES = {\n"; +for (my $i = 0; $i < @partition; $i += 3) { + my $end = $i + 2 < $#partition ? $i + 2 : $#partition; + my @items = map { sprintf("0x%X, 0x%X, %d,", @$_) } + @partition[$i .. $end]; + print " ", join(' ', @items), "\n"; +} +print " };\n\n private static final String[] ALIAS_KEYS = {\n"; +for (my $i = 0; $i < @alias_keys; $i += 6) { + my $end = $i + 5 < $#alias_keys ? $i + 5 : $#alias_keys; + print " ", join(', ', map { qq{\"$alias_keys[$_]\"} } $i .. $end), ",\n"; +} +print " };\n\n private static final short[] ALIAS_VALUE_IDS = {\n"; +for (my $i = 0; $i < @alias_keys; $i += 20) { + my $end = $i + 19 < $#alias_keys ? $i + 19 : $#alias_keys; + print " ", join(', ', map { $alias_value_id{$alias_keys[$_]} } $i .. $end), ",\n"; +} +print <<'FOOTER'; + }; + + private static final UnicodeSet[] SETS = buildSets(); + + static int valueCount() { + return VALUE_NAMES.length; + } + + static int aliasCount() { + return ALIAS_KEYS.length; + } + + static int rangeCount() { + return RANGES.length / 3; + } + + static String canonicalValue(int valueId) { + return VALUE_NAMES[valueId]; + } + + static UnicodeSet set(int valueId) { + return SETS[valueId]; + } + + static UnicodeSet set(String valueAlias) { + short valueId = value(valueAlias); + return valueId == INVALID ? null : SETS[valueId]; + } + + static short value(String valueAlias) { + String key = loose(valueAlias); + int index = Arrays.binarySearch(ALIAS_KEYS, key); + return index < 0 ? INVALID : ALIAS_VALUE_IDS[index]; + } + + static boolean isPropertyAlias(String alias) { + boolean hasIsPrefix = alias != null && alias.startsWith("Is"); + String normalized = loose(hasIsPrefix ? alias.substring(2) : alias); + return normalized.equals("blk") || normalized.equals("block"); + } + + private static UnicodeSet[] buildSets() { + UnicodeSet[] sets = new UnicodeSet[VALUE_NAMES.length]; + for (int valueId = 0; valueId < sets.length; valueId++) { + sets[valueId] = new UnicodeSet(); + } + for (int offset = 0; offset < RANGES.length; offset += 3) { + sets[RANGES[offset + 2]].add(RANGES[offset], RANGES[offset + 1]); + } + for (int valueId = 0; valueId < sets.length; valueId++) { + sets[valueId].freeze(); + } + return sets; + } + + private static String loose(String alias) { + if (alias == null) return ""; + StringBuilder normalized = new StringBuilder(alias.length()); + for (int i = 0; i < alias.length(); i++) { + char character = alias.charAt(i); + if (character == '_' || character == '-' || character == ' ' + || (character >= '\t' && character <= '\r')) continue; + normalized.append(Character.toLowerCase(character)); + } + return normalized.toString(); + } + + private PerlUnicodeBlockData() { + } +} +FOOTER diff --git a/docs/reference/feature-matrix.md b/docs/reference/feature-matrix.md index 806762290d..c2b756ed7c 100644 --- a/docs/reference/feature-matrix.md +++ b/docs/reference/feature-matrix.md @@ -379,7 +379,7 @@ my @copy = @{$z}; # ERROR - ✅ **Backreferences to Named Groups**: Using `\k` or `\g{name}` for backreferences to named groups is supported. - ✅ **Relative Backreferences**: Using `\g{-n}` for relative backreferences. - ✅ **Basic Unicode Properties**: Common `\p{...}` and `\P{...}` forms such as `\p{L}` execute through Joni. -- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script/Block aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, Numeric_Value, and Joining_Group assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, reserved or composite values, exact rationals, and Perl's generated decimal keyword aliases; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. +- 🟡 **Perl Unicode Property Syntax**: Perl-specific properties and focused Script aliases execute through Joni. `Age`, cumulative `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, Numeric_Value, Joining_Group, and Block assignments are generated from pinned Perl 5.44 Unicode 17.0 data with loose and wildcard aliases, ordered missing defaults, official compact Block aliases and shortcuts, reserved or composite values, exact rationals, and Perl's generated decimal keyword aliases; `ASCII_Hex_Digit`/`AHex` accepts Perl's eight boolean value aliases. Other generated property/value aliases still have pinned acceptance/rejection gaps. - ✅ **Possessive Quantifiers**: Quantifiers like `*+`, `++`, `?+`, and `{n,m}+`, which disable backtracking, are supported. - ✅ **Atomic Grouping**: Use of `(?>...)` for atomic groups is supported. - ✅ **`\K` assertion**: Keep left — in `s///`, text before `\K` is preserved; match variables reflect only the portion after `\K`. @@ -398,7 +398,7 @@ my @copy = @{$z}; # ERROR - ✅ **Branch Reset Groups**: `(?|...)` resets capture numbering across alternatives and preserves mapped match variables. - ✅ **Advanced Subroutine Calls**: Sub-pattern calls with numbered or named references like `(?1)` and `(?&name)` execute through Joni. - ✅ **Conditional Expressions**: Numbered and named capture conditions, positive and negative assertion conditions, recursion conditions `(?(R))`, `(?(R1))`, and `(?(R&name))`, executable callback conditions, and optimistic predicates execute through Joni. -- 🟡 **Extended Unicode Regex Features**: Focused Script, Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, Numeric_Value, Joining_Group, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 361,138/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. +- 🟡 **Extended Unicode Regex Features**: Focused Script, pinned Block, Script Extensions, `Extended_Pictographic`, `Age`, `In`/`Present_In`, General_Category, Canonical_Combining_Class, Bidi_Class, Decomposition_Type, East_Asian_Width, Numeric_Value, Joining_Group, and invalid-property gates execute through Joni, and `regexp_unicode_prop.t` passes 1,110/1,110. Current generated evidence is 369,462/407,367: the complete boundary corpus passes, while chunks 01–04 expose the remaining property/value alias gaps. - ✅ **Extended Grapheme Clusters**: Native `\b{gcb}`/`\B{gcb}` implement GB1–GB13 and GB999, and `\X` includes repeated GB9c Indic conjuncts. The complete 8,516-assertion GCB/`\X` section of authoritative chunk 05 passes identically on JVM and interpreter. - ✅ **Unicode Sentence Boundaries**: Native `\b{sb}`/`\B{sb}` implement SB1–SB11 and SB998 from the reproducible Perl 5.44 Unicode 17.0 sentence-break table. Authoritative chunk 05 passes 14,976/14,976 identically on JVM and interpreter. - ✅ **Unicode Word Boundaries**: Native `\b{wb}`/`\B{wb}` implement WB1–WB16 and WB999 from reproducible Perl 5.44 Unicode 17.0 word-break and extended-pictographic tables. Authoritative chunk 10 passes 19,510/19,510 identically on JVM and interpreter. diff --git a/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeBlockData.java b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeBlockData.java new file mode 100644 index 0000000000..011c23cd8a --- /dev/null +++ b/src/main/java/org/perlonjava/runtime/regex/PerlUnicodeBlockData.java @@ -0,0 +1,399 @@ +package org.perlonjava.runtime.regex; + +import com.ibm.icu.text.UnicodeSet; +import java.util.Arrays; + +/* + * Generated from Perl 5.44's pinned Unicode Character Database by + * dev/tools/generate_perl_unicode_block_data.pl. Do not edit manually. + * + * Unicode data source copyright: + * © 2025 Unicode®, Inc. + * Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in + * the U.S. and other countries. + * For terms of use and license, see https://www.unicode.org/terms_of_use.html + */ +final class PerlUnicodeBlockData { + static final String UNICODE_VERSION = "17.0.0"; + static final short INVALID = -1; + static final short NO_BLOCK = 0; + + private static final String[] VALUE_NAMES = { + "No_Block", "Basic_Latin", "Latin_1_Supplement", "Latin_Extended_A", "Latin_Extended_B", "IPA_Extensions", + "Spacing_Modifier_Letters", "Combining_Diacritical_Marks", "Greek_And_Coptic", "Cyrillic", "Cyrillic_Supplement", "Armenian", + "Hebrew", "Arabic", "Syriac", "Arabic_Supplement", "Thaana", "NKo", + "Samaritan", "Mandaic", "Syriac_Supplement", "Arabic_Extended_B", "Arabic_Extended_A", "Devanagari", + "Bengali", "Gurmukhi", "Gujarati", "Oriya", "Tamil", "Telugu", + "Kannada", "Malayalam", "Sinhala", "Thai", "Lao", "Tibetan", + "Myanmar", "Georgian", "Hangul_Jamo", "Ethiopic", "Ethiopic_Supplement", "Cherokee", + "Unified_Canadian_Aboriginal_Syllabics", "Ogham", "Runic", "Tagalog", "Hanunoo", "Buhid", + "Tagbanwa", "Khmer", "Mongolian", "Unified_Canadian_Aboriginal_Syllabics_Extended", "Limbu", "Tai_Le", + "New_Tai_Lue", "Khmer_Symbols", "Buginese", "Tai_Tham", "Combining_Diacritical_Marks_Extended", "Balinese", + "Sundanese", "Batak", "Lepcha", "Ol_Chiki", "Cyrillic_Extended_C", "Georgian_Extended", + "Sundanese_Supplement", "Vedic_Extensions", "Phonetic_Extensions", "Phonetic_Extensions_Supplement", "Combining_Diacritical_Marks_Supplement", "Latin_Extended_Additional", + "Greek_Extended", "General_Punctuation", "Superscripts_And_Subscripts", "Currency_Symbols", "Combining_Diacritical_Marks_For_Symbols", "Letterlike_Symbols", + "Number_Forms", "Arrows", "Mathematical_Operators", "Miscellaneous_Technical", "Control_Pictures", "Optical_Character_Recognition", + "Enclosed_Alphanumerics", "Box_Drawing", "Block_Elements", "Geometric_Shapes", "Miscellaneous_Symbols", "Dingbats", + "Miscellaneous_Mathematical_Symbols_A", "Supplemental_Arrows_A", "Braille_Patterns", "Supplemental_Arrows_B", "Miscellaneous_Mathematical_Symbols_B", "Supplemental_Mathematical_Operators", + "Miscellaneous_Symbols_And_Arrows", "Glagolitic", "Latin_Extended_C", "Coptic", "Georgian_Supplement", "Tifinagh", + "Ethiopic_Extended", "Cyrillic_Extended_A", "Supplemental_Punctuation", "CJK_Radicals_Supplement", "Kangxi_Radicals", "Ideographic_Description_Characters", + "CJK_Symbols_And_Punctuation", "Hiragana", "Katakana", "Bopomofo", "Hangul_Compatibility_Jamo", "Kanbun", + "Bopomofo_Extended", "CJK_Strokes", "Katakana_Phonetic_Extensions", "Enclosed_CJK_Letters_And_Months", "CJK_Compatibility", "CJK_Unified_Ideographs_Extension_A", + "Yijing_Hexagram_Symbols", "CJK_Unified_Ideographs", "Yi_Syllables", "Yi_Radicals", "Lisu", "Vai", + "Cyrillic_Extended_B", "Bamum", "Modifier_Tone_Letters", "Latin_Extended_D", "Syloti_Nagri", "Common_Indic_Number_Forms", + "Phags_Pa", "Saurashtra", "Devanagari_Extended", "Kayah_Li", "Rejang", "Hangul_Jamo_Extended_A", + "Javanese", "Myanmar_Extended_B", "Cham", "Myanmar_Extended_A", "Tai_Viet", "Meetei_Mayek_Extensions", + "Ethiopic_Extended_A", "Latin_Extended_E", "Cherokee_Supplement", "Meetei_Mayek", "Hangul_Syllables", "Hangul_Jamo_Extended_B", + "High_Surrogates", "High_Private_Use_Surrogates", "Low_Surrogates", "Private_Use_Area", "CJK_Compatibility_Ideographs", "Alphabetic_Presentation_Forms", + "Arabic_Presentation_Forms_A", "Variation_Selectors", "Vertical_Forms", "Combining_Half_Marks", "CJK_Compatibility_Forms", "Small_Form_Variants", + "Arabic_Presentation_Forms_B", "Halfwidth_And_Fullwidth_Forms", "Specials", "Linear_B_Syllabary", "Linear_B_Ideograms", "Aegean_Numbers", + "Ancient_Greek_Numbers", "Ancient_Symbols", "Phaistos_Disc", "Lycian", "Carian", "Coptic_Epact_Numbers", + "Old_Italic", "Gothic", "Old_Permic", "Ugaritic", "Old_Persian", "Deseret", + "Shavian", "Osmanya", "Osage", "Elbasan", "Caucasian_Albanian", "Vithkuqi", + "Todhri", "Linear_A", "Latin_Extended_F", "Cypriot_Syllabary", "Imperial_Aramaic", "Palmyrene", + "Nabataean", "Hatran", "Phoenician", "Lydian", "Sidetic", "Meroitic_Hieroglyphs", + "Meroitic_Cursive", "Kharoshthi", "Old_South_Arabian", "Old_North_Arabian", "Manichaean", "Avestan", + "Inscriptional_Parthian", "Inscriptional_Pahlavi", "Psalter_Pahlavi", "Old_Turkic", "Old_Hungarian", "Hanifi_Rohingya", + "Garay", "Rumi_Numeral_Symbols", "Yezidi", "Arabic_Extended_C", "Old_Sogdian", "Sogdian", + "Old_Uyghur", "Chorasmian", "Elymaic", "Brahmi", "Kaithi", "Sora_Sompeng", + "Chakma", "Mahajani", "Sharada", "Sinhala_Archaic_Numbers", "Khojki", "Multani", + "Khudawadi", "Grantha", "Tulu_Tigalari", "Newa", "Tirhuta", "Siddham", + "Modi", "Mongolian_Supplement", "Takri", "Myanmar_Extended_C", "Ahom", "Dogra", + "Warang_Citi", "Dives_Akuru", "Nandinagari", "Zanabazar_Square", "Soyombo", "Unified_Canadian_Aboriginal_Syllabics_Extended_A", + "Pau_Cin_Hau", "Devanagari_Extended_A", "Sharada_Supplement", "Sunuwar", "Bhaiksuki", "Marchen", + "Masaram_Gondi", "Gunjala_Gondi", "Tolong_Siki", "Makasar", "Kawi", "Lisu_Supplement", + "Tamil_Supplement", "Cuneiform", "Cuneiform_Numbers_And_Punctuation", "Early_Dynastic_Cuneiform", "Cypro_Minoan", "Egyptian_Hieroglyphs", + "Egyptian_Hieroglyph_Format_Controls", "Egyptian_Hieroglyphs_Extended_A", "Anatolian_Hieroglyphs", "Gurung_Khema", "Bamum_Supplement", "Mro", + "Tangsa", "Bassa_Vah", "Pahawh_Hmong", "Kirat_Rai", "Medefaidrin", "Beria_Erfe", + "Miao", "Ideographic_Symbols_And_Punctuation", "Tangut", "Tangut_Components", "Khitan_Small_Script", "Tangut_Supplement", + "Tangut_Components_Supplement", "Kana_Extended_B", "Kana_Supplement", "Kana_Extended_A", "Small_Kana_Extension", "Nushu", + "Duployan", "Shorthand_Format_Controls", "Symbols_For_Legacy_Computing_Supplement", "Miscellaneous_Symbols_Supplement", "Znamenny_Musical_Notation", "Byzantine_Musical_Symbols", + "Musical_Symbols", "Ancient_Greek_Musical_Notation", "Kaktovik_Numerals", "Mayan_Numerals", "Tai_Xuan_Jing_Symbols", "Counting_Rod_Numerals", + "Mathematical_Alphanumeric_Symbols", "Sutton_SignWriting", "Latin_Extended_G", "Glagolitic_Supplement", "Cyrillic_Extended_D", "Nyiakeng_Puachue_Hmong", + "Toto", "Wancho", "Nag_Mundari", "Ol_Onal", "Tai_Yo", "Ethiopic_Extended_B", + "Mende_Kikakui", "Adlam", "Indic_Siyaq_Numbers", "Ottoman_Siyaq_Numbers", "Arabic_Mathematical_Alphabetic_Symbols", "Mahjong_Tiles", + "Domino_Tiles", "Playing_Cards", "Enclosed_Alphanumeric_Supplement", "Enclosed_Ideographic_Supplement", "Miscellaneous_Symbols_And_Pictographs", "Emoticons", + "Ornamental_Dingbats", "Transport_And_Map_Symbols", "Alchemical_Symbols", "Geometric_Shapes_Extended", "Supplemental_Arrows_C", "Supplemental_Symbols_And_Pictographs", + "Chess_Symbols", "Symbols_And_Pictographs_Extended_A", "Symbols_For_Legacy_Computing", "CJK_Unified_Ideographs_Extension_B", "CJK_Unified_Ideographs_Extension_C", "CJK_Unified_Ideographs_Extension_D", + "CJK_Unified_Ideographs_Extension_E", "CJK_Unified_Ideographs_Extension_F", "CJK_Unified_Ideographs_Extension_I", "CJK_Compatibility_Ideographs_Supplement", "CJK_Unified_Ideographs_Extension_G", "CJK_Unified_Ideographs_Extension_H", + "CJK_Unified_Ideographs_Extension_J", "Tags", "Variation_Selectors_Supplement", "Supplementary_Private_Use_Area_A", "Supplementary_Private_Use_Area_B", + }; + + private static final int[] RANGES = { + 0x0, 0x7F, 1, 0x80, 0xFF, 2, 0x100, 0x17F, 3, + 0x180, 0x24F, 4, 0x250, 0x2AF, 5, 0x2B0, 0x2FF, 6, + 0x300, 0x36F, 7, 0x370, 0x3FF, 8, 0x400, 0x4FF, 9, + 0x500, 0x52F, 10, 0x530, 0x58F, 11, 0x590, 0x5FF, 12, + 0x600, 0x6FF, 13, 0x700, 0x74F, 14, 0x750, 0x77F, 15, + 0x780, 0x7BF, 16, 0x7C0, 0x7FF, 17, 0x800, 0x83F, 18, + 0x840, 0x85F, 19, 0x860, 0x86F, 20, 0x870, 0x89F, 21, + 0x8A0, 0x8FF, 22, 0x900, 0x97F, 23, 0x980, 0x9FF, 24, + 0xA00, 0xA7F, 25, 0xA80, 0xAFF, 26, 0xB00, 0xB7F, 27, + 0xB80, 0xBFF, 28, 0xC00, 0xC7F, 29, 0xC80, 0xCFF, 30, + 0xD00, 0xD7F, 31, 0xD80, 0xDFF, 32, 0xE00, 0xE7F, 33, + 0xE80, 0xEFF, 34, 0xF00, 0xFFF, 35, 0x1000, 0x109F, 36, + 0x10A0, 0x10FF, 37, 0x1100, 0x11FF, 38, 0x1200, 0x137F, 39, + 0x1380, 0x139F, 40, 0x13A0, 0x13FF, 41, 0x1400, 0x167F, 42, + 0x1680, 0x169F, 43, 0x16A0, 0x16FF, 44, 0x1700, 0x171F, 45, + 0x1720, 0x173F, 46, 0x1740, 0x175F, 47, 0x1760, 0x177F, 48, + 0x1780, 0x17FF, 49, 0x1800, 0x18AF, 50, 0x18B0, 0x18FF, 51, + 0x1900, 0x194F, 52, 0x1950, 0x197F, 53, 0x1980, 0x19DF, 54, + 0x19E0, 0x19FF, 55, 0x1A00, 0x1A1F, 56, 0x1A20, 0x1AAF, 57, + 0x1AB0, 0x1AFF, 58, 0x1B00, 0x1B7F, 59, 0x1B80, 0x1BBF, 60, + 0x1BC0, 0x1BFF, 61, 0x1C00, 0x1C4F, 62, 0x1C50, 0x1C7F, 63, + 0x1C80, 0x1C8F, 64, 0x1C90, 0x1CBF, 65, 0x1CC0, 0x1CCF, 66, + 0x1CD0, 0x1CFF, 67, 0x1D00, 0x1D7F, 68, 0x1D80, 0x1DBF, 69, + 0x1DC0, 0x1DFF, 70, 0x1E00, 0x1EFF, 71, 0x1F00, 0x1FFF, 72, + 0x2000, 0x206F, 73, 0x2070, 0x209F, 74, 0x20A0, 0x20CF, 75, + 0x20D0, 0x20FF, 76, 0x2100, 0x214F, 77, 0x2150, 0x218F, 78, + 0x2190, 0x21FF, 79, 0x2200, 0x22FF, 80, 0x2300, 0x23FF, 81, + 0x2400, 0x243F, 82, 0x2440, 0x245F, 83, 0x2460, 0x24FF, 84, + 0x2500, 0x257F, 85, 0x2580, 0x259F, 86, 0x25A0, 0x25FF, 87, + 0x2600, 0x26FF, 88, 0x2700, 0x27BF, 89, 0x27C0, 0x27EF, 90, + 0x27F0, 0x27FF, 91, 0x2800, 0x28FF, 92, 0x2900, 0x297F, 93, + 0x2980, 0x29FF, 94, 0x2A00, 0x2AFF, 95, 0x2B00, 0x2BFF, 96, + 0x2C00, 0x2C5F, 97, 0x2C60, 0x2C7F, 98, 0x2C80, 0x2CFF, 99, + 0x2D00, 0x2D2F, 100, 0x2D30, 0x2D7F, 101, 0x2D80, 0x2DDF, 102, + 0x2DE0, 0x2DFF, 103, 0x2E00, 0x2E7F, 104, 0x2E80, 0x2EFF, 105, + 0x2F00, 0x2FDF, 106, 0x2FE0, 0x2FEF, 0, 0x2FF0, 0x2FFF, 107, + 0x3000, 0x303F, 108, 0x3040, 0x309F, 109, 0x30A0, 0x30FF, 110, + 0x3100, 0x312F, 111, 0x3130, 0x318F, 112, 0x3190, 0x319F, 113, + 0x31A0, 0x31BF, 114, 0x31C0, 0x31EF, 115, 0x31F0, 0x31FF, 116, + 0x3200, 0x32FF, 117, 0x3300, 0x33FF, 118, 0x3400, 0x4DBF, 119, + 0x4DC0, 0x4DFF, 120, 0x4E00, 0x9FFF, 121, 0xA000, 0xA48F, 122, + 0xA490, 0xA4CF, 123, 0xA4D0, 0xA4FF, 124, 0xA500, 0xA63F, 125, + 0xA640, 0xA69F, 126, 0xA6A0, 0xA6FF, 127, 0xA700, 0xA71F, 128, + 0xA720, 0xA7FF, 129, 0xA800, 0xA82F, 130, 0xA830, 0xA83F, 131, + 0xA840, 0xA87F, 132, 0xA880, 0xA8DF, 133, 0xA8E0, 0xA8FF, 134, + 0xA900, 0xA92F, 135, 0xA930, 0xA95F, 136, 0xA960, 0xA97F, 137, + 0xA980, 0xA9DF, 138, 0xA9E0, 0xA9FF, 139, 0xAA00, 0xAA5F, 140, + 0xAA60, 0xAA7F, 141, 0xAA80, 0xAADF, 142, 0xAAE0, 0xAAFF, 143, + 0xAB00, 0xAB2F, 144, 0xAB30, 0xAB6F, 145, 0xAB70, 0xABBF, 146, + 0xABC0, 0xABFF, 147, 0xAC00, 0xD7AF, 148, 0xD7B0, 0xD7FF, 149, + 0xD800, 0xDB7F, 150, 0xDB80, 0xDBFF, 151, 0xDC00, 0xDFFF, 152, + 0xE000, 0xF8FF, 153, 0xF900, 0xFAFF, 154, 0xFB00, 0xFB4F, 155, + 0xFB50, 0xFDFF, 156, 0xFE00, 0xFE0F, 157, 0xFE10, 0xFE1F, 158, + 0xFE20, 0xFE2F, 159, 0xFE30, 0xFE4F, 160, 0xFE50, 0xFE6F, 161, + 0xFE70, 0xFEFF, 162, 0xFF00, 0xFFEF, 163, 0xFFF0, 0xFFFF, 164, + 0x10000, 0x1007F, 165, 0x10080, 0x100FF, 166, 0x10100, 0x1013F, 167, + 0x10140, 0x1018F, 168, 0x10190, 0x101CF, 169, 0x101D0, 0x101FF, 170, + 0x10200, 0x1027F, 0, 0x10280, 0x1029F, 171, 0x102A0, 0x102DF, 172, + 0x102E0, 0x102FF, 173, 0x10300, 0x1032F, 174, 0x10330, 0x1034F, 175, + 0x10350, 0x1037F, 176, 0x10380, 0x1039F, 177, 0x103A0, 0x103DF, 178, + 0x103E0, 0x103FF, 0, 0x10400, 0x1044F, 179, 0x10450, 0x1047F, 180, + 0x10480, 0x104AF, 181, 0x104B0, 0x104FF, 182, 0x10500, 0x1052F, 183, + 0x10530, 0x1056F, 184, 0x10570, 0x105BF, 185, 0x105C0, 0x105FF, 186, + 0x10600, 0x1077F, 187, 0x10780, 0x107BF, 188, 0x107C0, 0x107FF, 0, + 0x10800, 0x1083F, 189, 0x10840, 0x1085F, 190, 0x10860, 0x1087F, 191, + 0x10880, 0x108AF, 192, 0x108B0, 0x108DF, 0, 0x108E0, 0x108FF, 193, + 0x10900, 0x1091F, 194, 0x10920, 0x1093F, 195, 0x10940, 0x1095F, 196, + 0x10960, 0x1097F, 0, 0x10980, 0x1099F, 197, 0x109A0, 0x109FF, 198, + 0x10A00, 0x10A5F, 199, 0x10A60, 0x10A7F, 200, 0x10A80, 0x10A9F, 201, + 0x10AA0, 0x10ABF, 0, 0x10AC0, 0x10AFF, 202, 0x10B00, 0x10B3F, 203, + 0x10B40, 0x10B5F, 204, 0x10B60, 0x10B7F, 205, 0x10B80, 0x10BAF, 206, + 0x10BB0, 0x10BFF, 0, 0x10C00, 0x10C4F, 207, 0x10C50, 0x10C7F, 0, + 0x10C80, 0x10CFF, 208, 0x10D00, 0x10D3F, 209, 0x10D40, 0x10D8F, 210, + 0x10D90, 0x10E5F, 0, 0x10E60, 0x10E7F, 211, 0x10E80, 0x10EBF, 212, + 0x10EC0, 0x10EFF, 213, 0x10F00, 0x10F2F, 214, 0x10F30, 0x10F6F, 215, + 0x10F70, 0x10FAF, 216, 0x10FB0, 0x10FDF, 217, 0x10FE0, 0x10FFF, 218, + 0x11000, 0x1107F, 219, 0x11080, 0x110CF, 220, 0x110D0, 0x110FF, 221, + 0x11100, 0x1114F, 222, 0x11150, 0x1117F, 223, 0x11180, 0x111DF, 224, + 0x111E0, 0x111FF, 225, 0x11200, 0x1124F, 226, 0x11250, 0x1127F, 0, + 0x11280, 0x112AF, 227, 0x112B0, 0x112FF, 228, 0x11300, 0x1137F, 229, + 0x11380, 0x113FF, 230, 0x11400, 0x1147F, 231, 0x11480, 0x114DF, 232, + 0x114E0, 0x1157F, 0, 0x11580, 0x115FF, 233, 0x11600, 0x1165F, 234, + 0x11660, 0x1167F, 235, 0x11680, 0x116CF, 236, 0x116D0, 0x116FF, 237, + 0x11700, 0x1174F, 238, 0x11750, 0x117FF, 0, 0x11800, 0x1184F, 239, + 0x11850, 0x1189F, 0, 0x118A0, 0x118FF, 240, 0x11900, 0x1195F, 241, + 0x11960, 0x1199F, 0, 0x119A0, 0x119FF, 242, 0x11A00, 0x11A4F, 243, + 0x11A50, 0x11AAF, 244, 0x11AB0, 0x11ABF, 245, 0x11AC0, 0x11AFF, 246, + 0x11B00, 0x11B5F, 247, 0x11B60, 0x11B7F, 248, 0x11B80, 0x11BBF, 0, + 0x11BC0, 0x11BFF, 249, 0x11C00, 0x11C6F, 250, 0x11C70, 0x11CBF, 251, + 0x11CC0, 0x11CFF, 0, 0x11D00, 0x11D5F, 252, 0x11D60, 0x11DAF, 253, + 0x11DB0, 0x11DEF, 254, 0x11DF0, 0x11EDF, 0, 0x11EE0, 0x11EFF, 255, + 0x11F00, 0x11F5F, 256, 0x11F60, 0x11FAF, 0, 0x11FB0, 0x11FBF, 257, + 0x11FC0, 0x11FFF, 258, 0x12000, 0x123FF, 259, 0x12400, 0x1247F, 260, + 0x12480, 0x1254F, 261, 0x12550, 0x12F8F, 0, 0x12F90, 0x12FFF, 262, + 0x13000, 0x1342F, 263, 0x13430, 0x1345F, 264, 0x13460, 0x143FF, 265, + 0x14400, 0x1467F, 266, 0x14680, 0x160FF, 0, 0x16100, 0x1613F, 267, + 0x16140, 0x167FF, 0, 0x16800, 0x16A3F, 268, 0x16A40, 0x16A6F, 269, + 0x16A70, 0x16ACF, 270, 0x16AD0, 0x16AFF, 271, 0x16B00, 0x16B8F, 272, + 0x16B90, 0x16D3F, 0, 0x16D40, 0x16D7F, 273, 0x16D80, 0x16E3F, 0, + 0x16E40, 0x16E9F, 274, 0x16EA0, 0x16EDF, 275, 0x16EE0, 0x16EFF, 0, + 0x16F00, 0x16F9F, 276, 0x16FA0, 0x16FDF, 0, 0x16FE0, 0x16FFF, 277, + 0x17000, 0x187FF, 278, 0x18800, 0x18AFF, 279, 0x18B00, 0x18CFF, 280, + 0x18D00, 0x18D7F, 281, 0x18D80, 0x18DFF, 282, 0x18E00, 0x1AFEF, 0, + 0x1AFF0, 0x1AFFF, 283, 0x1B000, 0x1B0FF, 284, 0x1B100, 0x1B12F, 285, + 0x1B130, 0x1B16F, 286, 0x1B170, 0x1B2FF, 287, 0x1B300, 0x1BBFF, 0, + 0x1BC00, 0x1BC9F, 288, 0x1BCA0, 0x1BCAF, 289, 0x1BCB0, 0x1CBFF, 0, + 0x1CC00, 0x1CEBF, 290, 0x1CEC0, 0x1CEFF, 291, 0x1CF00, 0x1CFCF, 292, + 0x1CFD0, 0x1CFFF, 0, 0x1D000, 0x1D0FF, 293, 0x1D100, 0x1D1FF, 294, + 0x1D200, 0x1D24F, 295, 0x1D250, 0x1D2BF, 0, 0x1D2C0, 0x1D2DF, 296, + 0x1D2E0, 0x1D2FF, 297, 0x1D300, 0x1D35F, 298, 0x1D360, 0x1D37F, 299, + 0x1D380, 0x1D3FF, 0, 0x1D400, 0x1D7FF, 300, 0x1D800, 0x1DAAF, 301, + 0x1DAB0, 0x1DEFF, 0, 0x1DF00, 0x1DFFF, 302, 0x1E000, 0x1E02F, 303, + 0x1E030, 0x1E08F, 304, 0x1E090, 0x1E0FF, 0, 0x1E100, 0x1E14F, 305, + 0x1E150, 0x1E28F, 0, 0x1E290, 0x1E2BF, 306, 0x1E2C0, 0x1E2FF, 307, + 0x1E300, 0x1E4CF, 0, 0x1E4D0, 0x1E4FF, 308, 0x1E500, 0x1E5CF, 0, + 0x1E5D0, 0x1E5FF, 309, 0x1E600, 0x1E6BF, 0, 0x1E6C0, 0x1E6FF, 310, + 0x1E700, 0x1E7DF, 0, 0x1E7E0, 0x1E7FF, 311, 0x1E800, 0x1E8DF, 312, + 0x1E8E0, 0x1E8FF, 0, 0x1E900, 0x1E95F, 313, 0x1E960, 0x1EC6F, 0, + 0x1EC70, 0x1ECBF, 314, 0x1ECC0, 0x1ECFF, 0, 0x1ED00, 0x1ED4F, 315, + 0x1ED50, 0x1EDFF, 0, 0x1EE00, 0x1EEFF, 316, 0x1EF00, 0x1EFFF, 0, + 0x1F000, 0x1F02F, 317, 0x1F030, 0x1F09F, 318, 0x1F0A0, 0x1F0FF, 319, + 0x1F100, 0x1F1FF, 320, 0x1F200, 0x1F2FF, 321, 0x1F300, 0x1F5FF, 322, + 0x1F600, 0x1F64F, 323, 0x1F650, 0x1F67F, 324, 0x1F680, 0x1F6FF, 325, + 0x1F700, 0x1F77F, 326, 0x1F780, 0x1F7FF, 327, 0x1F800, 0x1F8FF, 328, + 0x1F900, 0x1F9FF, 329, 0x1FA00, 0x1FA6F, 330, 0x1FA70, 0x1FAFF, 331, + 0x1FB00, 0x1FBFF, 332, 0x1FC00, 0x1FFFF, 0, 0x20000, 0x2A6DF, 333, + 0x2A6E0, 0x2A6FF, 0, 0x2A700, 0x2B73F, 334, 0x2B740, 0x2B81F, 335, + 0x2B820, 0x2CEAF, 336, 0x2CEB0, 0x2EBEF, 337, 0x2EBF0, 0x2EE5F, 338, + 0x2EE60, 0x2F7FF, 0, 0x2F800, 0x2FA1F, 339, 0x2FA20, 0x2FFFF, 0, + 0x30000, 0x3134F, 340, 0x31350, 0x323AF, 341, 0x323B0, 0x3347F, 342, + 0x33480, 0xDFFFF, 0, 0xE0000, 0xE007F, 343, 0xE0080, 0xE00FF, 0, + 0xE0100, 0xE01EF, 344, 0xE01F0, 0xEFFFF, 0, 0xF0000, 0xFFFFF, 345, + 0x100000, 0x10FFFF, 346, + }; + + private static final String[] ALIAS_KEYS = { + "adlam", "aegeannumbers", "ahom", "alchemical", "alchemicalsymbols", "alphabeticpf", + "alphabeticpresentationforms", "anatolianhieroglyphs", "ancientgreekmusic", "ancientgreekmusicalnotation", "ancientgreeknumbers", "ancientsymbols", + "arabic", "arabicexta", "arabicextb", "arabicextc", "arabicextendeda", "arabicextendedb", + "arabicextendedc", "arabicmath", "arabicmathematicalalphabeticsymbols", "arabicpfa", "arabicpfb", "arabicpresentationformsa", + "arabicpresentationformsb", "arabicsup", "arabicsupplement", "armenian", "arrows", "ascii", + "avestan", "balinese", "bamum", "bamumsup", "bamumsupplement", "basiclatin", + "bassavah", "batak", "bengali", "beriaerfe", "bhaiksuki", "blockelements", + "bopomofo", "bopomofoext", "bopomofoextended", "boxdrawing", "brahmi", "braille", + "braillepatterns", "buginese", "buhid", "byzantinemusic", "byzantinemusicalsymbols", "canadiansyllabics", + "carian", "caucasianalbanian", "chakma", "cham", "cherokee", "cherokeesup", + "cherokeesupplement", "chesssymbols", "chorasmian", "cjk", "cjkcompat", "cjkcompatforms", + "cjkcompatibility", "cjkcompatibilityforms", "cjkcompatibilityideographs", "cjkcompatibilityideographssupplement", "cjkcompatideographs", "cjkcompatideographssup", + "cjkexta", "cjkextb", "cjkextc", "cjkextd", "cjkexte", "cjkextf", + "cjkextg", "cjkexth", "cjkexti", "cjkextj", "cjkradicalssup", "cjkradicalssupplement", + "cjkstrokes", "cjksymbols", "cjksymbolsandpunctuation", "cjkunifiedideographs", "cjkunifiedideographsextensiona", "cjkunifiedideographsextensionb", + "cjkunifiedideographsextensionc", "cjkunifiedideographsextensiond", "cjkunifiedideographsextensione", "cjkunifiedideographsextensionf", "cjkunifiedideographsextensiong", "cjkunifiedideographsextensionh", + "cjkunifiedideographsextensioni", "cjkunifiedideographsextensionj", "combiningdiacriticalmarks", "combiningdiacriticalmarksextended", "combiningdiacriticalmarksforsymbols", "combiningdiacriticalmarkssupplement", + "combininghalfmarks", "combiningmarksforsymbols", "commonindicnumberforms", "compatjamo", "controlpictures", "coptic", + "copticepactnumbers", "countingrod", "countingrodnumerals", "cuneiform", "cuneiformnumbers", "cuneiformnumbersandpunctuation", + "currencysymbols", "cypriotsyllabary", "cyprominoan", "cyrillic", "cyrillicexta", "cyrillicextb", + "cyrillicextc", "cyrillicextd", "cyrillicextendeda", "cyrillicextendedb", "cyrillicextendedc", "cyrillicextendedd", + "cyrillicsup", "cyrillicsupplement", "cyrillicsupplementary", "deseret", "devanagari", "devanagariext", + "devanagariexta", "devanagariextended", "devanagariextendeda", "diacriticals", "diacriticalsext", "diacriticalsforsymbols", + "diacriticalssup", "dingbats", "divesakuru", "dogra", "domino", "dominotiles", + "duployan", "earlydynasticcuneiform", "egyptianhieroglyphformatcontrols", "egyptianhieroglyphs", "egyptianhieroglyphsexta", "egyptianhieroglyphsextendeda", + "elbasan", "elymaic", "emoticons", "enclosedalphanum", "enclosedalphanumerics", "enclosedalphanumericsupplement", + "enclosedalphanumsup", "enclosedcjk", "enclosedcjklettersandmonths", "enclosedideographicsup", "enclosedideographicsupplement", "ethiopic", + "ethiopicext", "ethiopicexta", "ethiopicextb", "ethiopicextended", "ethiopicextendeda", "ethiopicextendedb", + "ethiopicsup", "ethiopicsupplement", "garay", "generalpunctuation", "geometricshapes", "geometricshapesext", + "geometricshapesextended", "georgian", "georgianext", "georgianextended", "georgiansup", "georgiansupplement", + "glagolitic", "glagoliticsup", "glagoliticsupplement", "gothic", "grantha", "greek", + "greekandcoptic", "greekext", "greekextended", "gujarati", "gunjalagondi", "gurmukhi", + "gurungkhema", "halfandfullforms", "halfmarks", "halfwidthandfullwidthforms", "hangul", "hangulcompatibilityjamo", + "hanguljamo", "hanguljamoextendeda", "hanguljamoextendedb", "hangulsyllables", "hanifirohingya", "hanunoo", + "hatran", "hebrew", "highprivateusesurrogates", "highpusurrogates", "highsurrogates", "hiragana", + "idc", "ideographicdescriptioncharacters", "ideographicsymbols", "ideographicsymbolsandpunctuation", "imperialaramaic", "indicnumberforms", + "indicsiyaqnumbers", "inscriptionalpahlavi", "inscriptionalparthian", "ipaext", "ipaextensions", "jamo", + "jamoexta", "jamoextb", "javanese", "kaithi", "kaktoviknumerals", "kanaexta", + "kanaextb", "kanaextendeda", "kanaextendedb", "kanasup", "kanasupplement", "kanbun", + "kangxi", "kangxiradicals", "kannada", "katakana", "katakanaext", "katakanaphoneticextensions", + "kawi", "kayahli", "kharoshthi", "khitansmallscript", "khmer", "khmersymbols", + "khojki", "khudawadi", "kiratrai", "lao", "latin1", "latin1sup", + "latin1supplement", "latinexta", "latinextadditional", "latinextb", "latinextc", "latinextd", + "latinexte", "latinextendeda", "latinextendedadditional", "latinextendedb", "latinextendedc", "latinextendedd", + "latinextendede", "latinextendedf", "latinextendedg", "latinextf", "latinextg", "lepcha", + "letterlikesymbols", "limbu", "lineara", "linearbideograms", "linearbsyllabary", "lisu", + "lisusup", "lisusupplement", "lowsurrogates", "lycian", "lydian", "mahajani", + "mahjong", "mahjongtiles", "makasar", "malayalam", "mandaic", "manichaean", + "marchen", "masaramgondi", "mathalphanum", "mathematicalalphanumericsymbols", "mathematicaloperators", "mathoperators", + "mayannumerals", "medefaidrin", "meeteimayek", "meeteimayekext", "meeteimayekextensions", "mendekikakui", + "meroiticcursive", "meroitichieroglyphs", "miao", "miscarrows", "miscellaneousmathematicalsymbolsa", "miscellaneousmathematicalsymbolsb", + "miscellaneoussymbols", "miscellaneoussymbolsandarrows", "miscellaneoussymbolsandpictographs", "miscellaneoussymbolssupplement", "miscellaneoustechnical", "miscmathsymbolsa", + "miscmathsymbolsb", "miscpictographs", "miscsymbols", "miscsymbolssup", "misctechnical", "modi", + "modifierletters", "modifiertoneletters", "mongolian", "mongoliansup", "mongoliansupplement", "mro", + "multani", "music", "musicalsymbols", "myanmar", "myanmarexta", "myanmarextb", + "myanmarextc", "myanmarextendeda", "myanmarextendedb", "myanmarextendedc", "nabataean", "nagmundari", + "nandinagari", "nb", "newa", "newtailue", "nko", "noblock", + "numberforms", "nushu", "nyiakengpuachuehmong", "ocr", "ogham", "olchiki", + "oldhungarian", "olditalic", "oldnortharabian", "oldpermic", "oldpersian", "oldsogdian", + "oldsoutharabian", "oldturkic", "olduyghur", "olonal", "opticalcharacterrecognition", "oriya", + "ornamentaldingbats", "osage", "osmanya", "ottomansiyaqnumbers", "pahawhhmong", "palmyrene", + "paucinhau", "phagspa", "phaistos", "phaistosdisc", "phoenician", "phoneticext", + "phoneticextensions", "phoneticextensionssupplement", "phoneticextsup", "playingcards", "privateuse", "privateusearea", + "psalterpahlavi", "pua", "punctuation", "rejang", "rumi", "ruminumeralsymbols", + "runic", "samaritan", "saurashtra", "sharada", "sharadasup", "sharadasupplement", + "shavian", "shorthandformatcontrols", "siddham", "sidetic", "sinhala", "sinhalaarchaicnumbers", + "smallforms", "smallformvariants", "smallkanaext", "smallkanaextension", "sogdian", "sorasompeng", + "soyombo", "spacingmodifierletters", "specials", "sundanese", "sundanesesup", "sundanesesupplement", + "sunuwar", "suparrowsa", "suparrowsb", "suparrowsc", "superandsub", "superscriptsandsubscripts", + "supmathoperators", "supplementalarrowsa", "supplementalarrowsb", "supplementalarrowsc", "supplementalmathematicaloperators", "supplementalpunctuation", + "supplementalsymbolsandpictographs", "supplementaryprivateuseareaa", "supplementaryprivateuseareab", "suppuaa", "suppuab", "suppunctuation", + "supsymbolsandpictographs", "suttonsignwriting", "sylotinagri", "symbolsandpictographsexta", "symbolsandpictographsextendeda", "symbolsforlegacycomputing", + "symbolsforlegacycomputingsup", "symbolsforlegacycomputingsupplement", "syriac", "syriacsup", "syriacsupplement", "tagalog", + "tagbanwa", "tags", "taile", "taitham", "taiviet", "taixuanjing", + "taixuanjingsymbols", "taiyo", "takri", "tamil", "tamilsup", "tamilsupplement", + "tangsa", "tangut", "tangutcomponents", "tangutcomponentssup", "tangutcomponentssupplement", "tangutsup", + "tangutsupplement", "telugu", "thaana", "thai", "tibetan", "tifinagh", + "tirhuta", "todhri", "tolongsiki", "toto", "transportandmap", "transportandmapsymbols", + "tulutigalari", "ucas", "ucasext", "ucasexta", "ugaritic", "unifiedcanadianaboriginalsyllabics", + "unifiedcanadianaboriginalsyllabicsextended", "unifiedcanadianaboriginalsyllabicsextendeda", "vai", "variationselectors", "variationselectorssupplement", "vedicext", + "vedicextensions", "verticalforms", "vithkuqi", "vs", "vssup", "wancho", + "warangciti", "yezidi", "yijing", "yijinghexagramsymbols", "yiradicals", "yisyllables", + "zanabazarsquare", "znamennymusic", "znamennymusicalnotation", + }; + + private static final short[] ALIAS_VALUE_IDS = { + 313, 167, 238, 326, 326, 155, 155, 266, 295, 295, 168, 169, 13, 22, 21, 213, 22, 21, 213, 316, + 316, 156, 162, 156, 162, 15, 15, 11, 79, 1, 203, 59, 127, 268, 268, 1, 271, 61, 24, 275, + 250, 86, 111, 114, 114, 85, 219, 92, 92, 56, 47, 293, 293, 42, 172, 184, 222, 140, 41, 146, + 146, 330, 217, 121, 118, 160, 118, 160, 154, 339, 154, 339, 119, 333, 334, 335, 336, 337, 340, 341, + 338, 342, 105, 105, 115, 108, 108, 121, 119, 333, 334, 335, 336, 337, 340, 341, 338, 342, 7, 58, + 76, 70, 159, 76, 131, 112, 82, 99, 173, 299, 299, 259, 260, 260, 75, 189, 262, 9, 103, 126, + 64, 304, 103, 126, 64, 304, 10, 10, 10, 179, 23, 134, 247, 134, 247, 7, 58, 76, 70, 89, + 241, 239, 318, 318, 288, 261, 264, 263, 265, 265, 183, 218, 323, 84, 84, 320, 320, 117, 117, 321, + 321, 39, 102, 144, 311, 102, 144, 311, 40, 40, 210, 73, 87, 327, 327, 37, 65, 65, 100, 100, + 97, 303, 303, 175, 229, 8, 8, 72, 72, 26, 253, 25, 267, 163, 159, 163, 148, 112, 38, 137, + 149, 148, 209, 46, 193, 12, 151, 151, 150, 109, 107, 107, 277, 277, 190, 131, 314, 205, 204, 5, + 5, 38, 137, 149, 138, 220, 296, 285, 283, 285, 283, 284, 284, 113, 106, 106, 30, 110, 116, 116, + 256, 135, 199, 280, 49, 55, 226, 228, 273, 34, 2, 2, 2, 3, 71, 4, 98, 129, 145, 3, + 71, 4, 98, 129, 145, 188, 302, 188, 302, 62, 77, 52, 187, 166, 165, 124, 257, 257, 152, 171, + 195, 223, 317, 317, 255, 31, 19, 202, 251, 252, 300, 300, 80, 80, 297, 274, 147, 143, 143, 312, + 198, 197, 276, 96, 90, 94, 88, 96, 322, 291, 81, 90, 94, 322, 88, 291, 81, 234, 6, 128, + 50, 235, 235, 269, 227, 294, 294, 36, 141, 139, 237, 141, 139, 237, 192, 308, 242, 0, 231, 54, + 17, 0, 78, 287, 305, 83, 43, 63, 208, 174, 201, 176, 178, 214, 200, 207, 216, 309, 83, 27, + 324, 182, 181, 315, 272, 191, 246, 132, 170, 170, 194, 68, 68, 69, 69, 319, 153, 153, 206, 153, + 73, 136, 211, 211, 44, 18, 133, 224, 248, 248, 180, 289, 233, 196, 32, 225, 161, 161, 286, 286, + 215, 221, 244, 6, 164, 60, 66, 66, 249, 91, 93, 328, 74, 74, 95, 91, 93, 328, 95, 104, + 329, 345, 346, 345, 346, 104, 329, 301, 130, 331, 331, 332, 290, 290, 14, 20, 20, 45, 48, 343, + 53, 57, 142, 298, 298, 310, 236, 28, 258, 258, 270, 278, 279, 282, 282, 281, 281, 29, 16, 33, + 35, 101, 232, 186, 254, 306, 325, 325, 230, 42, 51, 245, 177, 42, 51, 245, 125, 157, 344, 67, + 67, 158, 185, 157, 344, 307, 240, 212, 120, 120, 123, 122, 243, 292, 292, + }; + + private static final UnicodeSet[] SETS = buildSets(); + + static int valueCount() { + return VALUE_NAMES.length; + } + + static int aliasCount() { + return ALIAS_KEYS.length; + } + + static int rangeCount() { + return RANGES.length / 3; + } + + static String canonicalValue(int valueId) { + return VALUE_NAMES[valueId]; + } + + static UnicodeSet set(int valueId) { + return SETS[valueId]; + } + + static UnicodeSet set(String valueAlias) { + short valueId = value(valueAlias); + return valueId == INVALID ? null : SETS[valueId]; + } + + static short value(String valueAlias) { + String key = loose(valueAlias); + int index = Arrays.binarySearch(ALIAS_KEYS, key); + return index < 0 ? INVALID : ALIAS_VALUE_IDS[index]; + } + + static boolean isPropertyAlias(String alias) { + boolean hasIsPrefix = alias != null && alias.startsWith("Is"); + String normalized = loose(hasIsPrefix ? alias.substring(2) : alias); + return normalized.equals("blk") || normalized.equals("block"); + } + + private static UnicodeSet[] buildSets() { + UnicodeSet[] sets = new UnicodeSet[VALUE_NAMES.length]; + for (int valueId = 0; valueId < sets.length; valueId++) { + sets[valueId] = new UnicodeSet(); + } + for (int offset = 0; offset < RANGES.length; offset += 3) { + sets[RANGES[offset + 2]].add(RANGES[offset], RANGES[offset + 1]); + } + for (int valueId = 0; valueId < sets.length; valueId++) { + sets[valueId].freeze(); + } + return sets; + } + + private static String loose(String alias) { + if (alias == null) return ""; + StringBuilder normalized = new StringBuilder(alias.length()); + for (int i = 0; i < alias.length(); i++) { + char character = alias.charAt(i); + if (character == '_' || character == '-' || character == ' ' + || (character >= '\t' && character <= '\r')) continue; + normalized.append(Character.toLowerCase(character)); + } + return normalized.toString(); + } + + private PerlUnicodeBlockData() { + } +} diff --git a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java index a74ccda49a..321131f13c 100644 --- a/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java +++ b/src/main/java/org/perlonjava/runtime/regex/UnicodeResolver.java @@ -701,6 +701,8 @@ private static String translateUnicodeProperty(String property, boolean negated, isPerlIsPrefixedNumericWildcard(property); boolean isPrefixedJoiningGroupWildcard = isPerlIsPrefixedJoiningGroupWildcard(property); + boolean isPrefixedBlockWildcard = + isPerlIsPrefixedBlockWildcard(property); property = normalizePerlIsPropertyAssignment(property); if (isPrefixedNumericWildcard) { throw new IllegalArgumentException( @@ -710,6 +712,10 @@ private static String translateUnicodeProperty(String property, boolean negated, throw new IllegalArgumentException( "Can't find Unicode property definition for Is-prefixed Joining_Group wildcard"); } + if (isPrefixedBlockWildcard) { + throw new IllegalArgumentException( + "Can't find Unicode property definition for Is-prefixed Block wildcard"); + } if (property.startsWith("utf8::")) { String userPropertyName = property.substring("utf8::".length()); if (!userPropertyName.matches("[A-Za-z_][A-Za-z0-9_]*")) { @@ -1029,6 +1035,11 @@ && isCanonicalCombiningClassProperty(alias.substring(0, assignment))) { alias.substring(0, assignment))) { return resolvePerlJoiningGroup(alias.substring(assignment + 1)); } + if (assignment > 0 && assignment < alias.length() - 1 + && PerlUnicodeBlockData.isPropertyAlias( + alias.substring(0, assignment))) { + return resolvePerlBlock(alias.substring(assignment + 1)); + } if (assignment > 0 && assignment < alias.length() - 1) { Boolean value = perlBooleanPropertyValue(alias.substring(assignment + 1)); if (value != null) { @@ -1041,7 +1052,8 @@ && loosePropertyName(alias.substring(0, assignment)) .applyPropertyAlias("ASCII_Hex_Digit", "True"); } if (binaryProperty != null) { - return value ? binaryProperty : binaryProperty.complement(); + return value ? binaryProperty + : new UnicodeSet(binaryProperty).complement().freeze(); } } } @@ -1056,6 +1068,8 @@ && loosePropertyName(alias.substring(0, assignment)) } String blockAlias = alias; + boolean blockShortcut = false; + boolean isBlockShortcut = false; if (alias.length() > 2 && alias.regionMatches(true, 0, "in", 0, 2)) { int valueStart = 2; while (valueStart < alias.length()) { @@ -1065,19 +1079,36 @@ && loosePropertyName(alias.substring(0, assignment)) } if (valueStart >= alias.length()) return null; blockAlias = alias.substring(valueStart); + blockShortcut = true; } else if (assignment >= 0 || unicodePropertyValue(UProperty.SCRIPT, alias) >= 0) { return null; - } - - int blockValue = unicodePropertyValue(UProperty.BLOCK, blockAlias); - if (blockValue < 0) return null; - String canonicalBlock = UCharacter.getPropertyValueName( - UProperty.BLOCK, blockValue, UProperty.NameChoice.LONG); - if (canonicalBlock == null || !loosePropertyName(blockAlias) - .equals(loosePropertyName(canonicalBlock))) { + } else if (alias.length() > 2 && alias.regionMatches(true, 0, "is", 0, 2)) { + int valueStart = 2; + while (valueStart < alias.length()) { + char separator = alias.charAt(valueStart); + if (!Character.isWhitespace(separator) + && separator != '-' && separator != '_') break; + valueStart++; + } + if (valueStart >= alias.length()) return null; + String candidate = alias.substring(valueStart); + if (unicodePropertyValue(UProperty.SCRIPT, candidate) >= 0) return null; + if (isIcuBinaryPropertyAlias(candidate) + || isIcuGeneralCategoryAlias(candidate)) return null; + blockAlias = candidate; + blockShortcut = true; + isBlockShortcut = true; + } + if (!blockShortcut && (isIcuBinaryPropertyAlias(blockAlias) + || isIcuGeneralCategoryAlias(blockAlias))) return null; + UnicodeSet block = PerlUnicodeBlockData.set(blockAlias); + if (isBlockShortcut && block != null && block.containsSome(0xD800, 0xDFFF)) { + // Joni's UTF-8 subject path cannot represent an isolated surrogate. + // Retain the established deferred single-Is behavior until that + // representation debt is closed; explicit Block=/In forms remain pinned. return null; } - return new UnicodeSet().applyIntPropertyValue(UProperty.BLOCK, blockValue); + return block; } private static Boolean perlBooleanPropertyValue(String value) { @@ -1088,6 +1119,24 @@ private static Boolean perlBooleanPropertyValue(String value) { }; } + private static boolean isIcuBinaryPropertyAlias(String alias) { + try { + new UnicodeSet().applyPropertyAlias(alias, "True"); + return true; + } catch (IllegalArgumentException unsupported) { + return false; + } + } + + private static boolean isIcuGeneralCategoryAlias(String alias) { + try { + new UnicodeSet().applyPropertyAlias("General_Category", alias); + return true; + } catch (IllegalArgumentException unsupported) { + return false; + } + } + private static boolean isPerlIsPrefixedNumericWildcard(String property) { int assignment = propertyValueDelimiter(property); if (assignment <= 0 || assignment == property.length() - 1) return false; @@ -1108,6 +1157,16 @@ private static boolean isPerlIsPrefixedJoiningGroupWildcard(String property) { && perlNumericWildcardBody(property.substring(assignment + 1)) != null; } + private static boolean isPerlIsPrefixedBlockWildcard(String property) { + int assignment = propertyValueDelimiter(property); + if (assignment <= 0 || assignment == property.length() - 1) return false; + String propertyName = exactIsPrefixedPropertyName( + property.substring(0, assignment)); + return propertyName != null + && PerlUnicodeBlockData.isPropertyAlias(propertyName) + && perlBlockWildcardBody(property.substring(assignment + 1)) != null; + } + private static String exactIsPrefixedPropertyName(String name) { if (name.length() <= 2 || name.charAt(0) != 'I' || name.charAt(1) != 's') { return null; @@ -1161,6 +1220,58 @@ private static UnicodeSet resolvePerlJoiningGroup(String value) { return result.freeze(); } + private static UnicodeSet resolvePerlBlock(String value) { + String wildcard = perlBlockWildcardBody(value); + if (wildcard == null) { + UnicodeSet exact = PerlUnicodeBlockData.set(value); + if (exact == null) { + throw new IllegalArgumentException( + "Unsupported Block value: " + value.trim()); + } + return exact; + } + if (wildcard.indexOf('*') >= 0) { + throw new IllegalArgumentException( + "quantifier '*' is not allowed in Unicode property value wildcard"); + } + + Pattern valuePattern; + try { + valuePattern = Pattern.compile(wildcard); + } catch (RuntimeException invalidPattern) { + throw new IllegalArgumentException( + "Invalid Unicode property value wildcard", invalidPattern); + } + + UnicodeSet result = new UnicodeSet(); + for (int valueId = 0; valueId < PerlUnicodeBlockData.valueCount(); valueId++) { + String candidate = PerlUnicodeBlockData.canonicalValue(valueId); + if (valuePattern.matcher(candidate).matches() + || valuePattern.matcher(loosePropertyName(candidate)).matches()) { + result.addAll(PerlUnicodeBlockData.set(valueId)); + } + } + if (result.isEmpty()) { + throw new IllegalArgumentException( + "No Unicode property value wildcard matches Block"); + } + return result.freeze(); + } + + private static String perlBlockWildcardBody(String value) { + String trimmed = value.trim(); + if (!trimmed.startsWith("#") || !trimmed.endsWith("#") + || trimmed.length() <= 2) { + return null; + } + String body = trimmed.substring(1, trimmed.length() - 1); + if (body.startsWith("\\A") && body.endsWith("\\z") + && body.length() > 4) { + return body.substring(2, body.length() - 2); + } + return body; + } + private static UnicodeSet resolvePerlNumericValue(String value) { String wildcard = perlNumericWildcardBody(value); if (wildcard != null) { diff --git a/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeBlockDataTest.java b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeBlockDataTest.java new file mode 100644 index 0000000000..887ea036ff --- /dev/null +++ b/src/test/java/org/perlonjava/runtime/regex/PerlUnicodeBlockDataTest.java @@ -0,0 +1,158 @@ +package org.perlonjava.runtime.regex; + +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import com.ibm.icu.text.UnicodeSet; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.HashSet; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; + +@Tag("unit") +class PerlUnicodeBlockDataTest { + private static final int MAX_CODE_POINT = 0x10ffff; + + @Test + void usesPinnedPerl544Unicode17Data() { + assertEquals("17.0.0", PerlUnicodeBlockData.UNICODE_VERSION); + assertEquals(347, PerlUnicodeBlockData.valueCount()); + assertEquals(495, PerlUnicodeBlockData.aliasCount()); + assertEquals(397, PerlUnicodeBlockData.rangeCount()); + assertEquals("No_Block", + PerlUnicodeBlockData.canonicalValue(PerlUnicodeBlockData.NO_BLOCK)); + + assertRange("Sidetic", 0x10940, 0x1095f); + assertRange("Sharada_Sup", 0x11b60, 0x11b7f); + assertRange("Tolong_Siki", 0x11db0, 0x11def); + assertRange("Beria_Erfe", 0x16ea0, 0x16edf); + assertRange("Tangut_Components_Sup", 0x18d80, 0x18dff); + assertRange("Misc_Symbols_Sup", 0x1cec0, 0x1ceff); + assertRange("Tai_Yo", 0x1e6c0, 0x1e6ff); + assertRange("CJK_Ext_J", 0x323b0, 0x3347f); + } + + @Test + void formsTheCompleteDisjointBlockPartition() { + UnicodeSet union = new UnicodeSet(); + Set names = new HashSet<>(); + int namedCodePoints = 0; + int rangeCount = 0; + + for (int valueId = 0; valueId < PerlUnicodeBlockData.valueCount(); valueId++) { + String name = PerlUnicodeBlockData.canonicalValue(valueId); + UnicodeSet set = PerlUnicodeBlockData.set(valueId); + assertTrue(names.add(name), name); + assertTrue(set.isFrozen(), name); + assertFalse(set.isEmpty(), name); + assertTrue(new UnicodeSet(union).retainAll(set).isEmpty(), name); + union.addAll(set); + rangeCount += set.getRangeCount(); + if (valueId != PerlUnicodeBlockData.NO_BLOCK) { + assertEquals(1, set.getRangeCount(), name); + namedCodePoints += set.size(); + } + } + + assertEquals(347, names.size()); + assertEquals(397, rangeCount); + assertEquals(303_808, namedCodePoints); + assertEquals(new UnicodeSet(0, MAX_CODE_POINT), union); + } + + @Test + void noBlockIsThePinnedDefaultComplement() { + UnicodeSet noBlock = PerlUnicodeBlockData.set(PerlUnicodeBlockData.NO_BLOCK); + assertEquals(51, noBlock.getRangeCount()); + assertEquals(810_304, noBlock.size()); + assertTrue(noBlock.contains(0x2fe0)); + assertTrue(noBlock.contains(0x33480)); + assertFalse(noBlock.contains(0x2f00)); + + assertTrue(set("Greek_And_Coptic").contains(0x0378), + "unassigned code points retain their enclosing block"); + assertTrue(set("Arabic_Presentation_Forms_A").contains(0xfdd0), + "noncharacters retain their enclosing block"); + assertTrue(set("Specials").contains(0xfffe)); + assertTrue(set("Sup_PUA_B").contains(MAX_CODE_POINT)); + } + + @Test + void resolvesEveryAliasFamilyWithPerlLooseMatching() { + assertSame(set("Basic_Latin"), set("ASCII")); + assertSame(set("Basic_Latin"), set("b-a s_i c_latin")); + assertSame(set("Latin_1_Supplement"), set("Latin1Sup")); + assertSame(set("Greek_And_Coptic"), set("Greek")); + assertSame(set("No_Block"), set("NB")); + + assertSame(set("Arabic_Presentation_Forms_A"), + set("Arabic_Presentation_Forms-A")); + assertSame(set("Unified_Canadian_Aboriginal_Syllabics"), + set("Canadian_Syllabics")); + assertSame(set("Combining_Diacritical_Marks_For_Symbols"), + set("Combining_Marks_For_Symbols")); + assertSame(set("Cyrillic_Supplement"), set("Cyrillic_Supplementary")); + assertSame(set("Latin_1_Supplement"), set("Latin_1")); + assertSame(set("Private_Use_Area"), set("Private_Use")); + + assertNull(PerlUnicodeBlockData.set("Definitely_Not_A_Block")); + assertEquals(PerlUnicodeBlockData.INVALID, + PerlUnicodeBlockData.value("Definitely_Not_A_Block")); + } + + @Test + void followsPerlLoosePropertyAliasRules() { + assertTrue(PerlUnicodeBlockData.isPropertyAlias("blk")); + assertTrue(PerlUnicodeBlockData.isPropertyAlias("Block")); + assertTrue(PerlUnicodeBlockData.isPropertyAlias("b_l-k")); + assertTrue(PerlUnicodeBlockData.isPropertyAlias("IsBlock")); + assertTrue(PerlUnicodeBlockData.isPropertyAlias("Is_Block")); + assertTrue(PerlUnicodeBlockData.isPropertyAlias("Isblock")); + + assertFalse(PerlUnicodeBlockData.isPropertyAlias("isBlock")); + assertFalse(PerlUnicodeBlockData.isPropertyAlias("ISBlock")); + assertFalse(PerlUnicodeBlockData.isPropertyAlias("Block\u1680")); + assertFalse(PerlUnicodeBlockData.isPropertyAlias("Blocks")); + assertFalse(PerlUnicodeBlockData.isPropertyAlias(null)); + } + + @Test + void generatorReproducesCheckedInDataByteForByte() throws Exception { + Path generator = Path.of("dev", "tools", "generate_perl_unicode_block_data.pl"); + Path generated = Path.of("src", "main", "java", "org", "perlonjava", "runtime", + "regex", "PerlUnicodeBlockData.java"); + Process process = new ProcessBuilder("perl", generator.toString()) + .redirectErrorStream(true) + .start(); + byte[] actual = process.getInputStream().readAllBytes(); + assertTrue(process.waitFor(30, TimeUnit.SECONDS), "generator timed out"); + assertEquals(0, process.exitValue(), new String(actual, StandardCharsets.UTF_8)); + assertArrayEquals(Files.readAllBytes(generated), actual); + for (byte value : actual) { + assertFalse(value == '\r', "generated source must use raw LF"); + } + } + + private static UnicodeSet set(String alias) { + UnicodeSet set = PerlUnicodeBlockData.set(alias); + if (set == null) throw new AssertionError("Unknown Block alias " + alias); + return set; + } + + private static void assertRange(String alias, int first, int last) { + UnicodeSet set = set(alias); + assertEquals(1, set.getRangeCount(), alias); + assertEquals(first, set.getRangeStart(0), alias); + assertEquals(last, set.getRangeEnd(0), alias); + assertFalse(set.contains(first - 1), alias); + assertFalse(set.contains(last + 1), alias); + } +} diff --git a/src/test/resources/unit/regex/unicode_block_data.t b/src/test/resources/unit/regex/unicode_block_data.t new file mode 100644 index 0000000000..e6f2bf306b --- /dev/null +++ b/src/test/resources/unit/regex/unicode_block_data.t @@ -0,0 +1,84 @@ +use strict; +use warnings; +use Test::More; +use Unicode::UCD (); + +no warnings 'experimental::uniprop_wildcards'; + +sub accepts_property { + my ($property, $cp, $name) = @_; + my $re = eval 'qr/\\A\\p{' . $property . '}\\z/u'; + my $error = $@; + my $matched = $re ? eval { chr($cp) =~ $re ? 1 : 0 } : 0; + $error ||= $@; + ok(!$error && $matched, $name) or diag("property=$property error=$error"); +} + +sub rejects_property { + my ($property, $name) = @_; + my $re = eval 'qr/\\A\\p{' . $property . '}\\z/u'; + my $error = $@; + eval { "A" =~ $re } if $re && !$error; + $error ||= $@; + ok($error, $name) or diag("unexpectedly accepted property=$property"); +} + +TODO: { + local $TODO = 'Block resolver wiring is a separate Phase 36 slice'; + + accepts_property('Block=Basic_Latin', 0x41, 'Block long property/value'); + accepts_property('Blk=ASCII', 0x41, 'Blk and ASCII official aliases'); + accepts_property('Blk=Latin1Sup', 0x80, 'official compact block value alias'); + accepts_property('b_l-k=b-a s_i c_latin', 0x41, + 'loose matching in name and value'); + accepts_property('Block: BasicLatin', 0x41, 'colon separator and loose value'); + + accepts_property('InBasicLatin', 0x41, 'InBlockName shortcut'); + accepts_property('INBasicLatin', 0x41, 'single-form In name is loosely matched'); + accepts_property('isBasicLatin', 0x41, 'single-form Is name is loosely matched'); + accepts_property('IsBlock=Basic_Latin', 0x41, 'exact Is compound prefix'); + accepts_property('Is_Block=Basic_Latin', 0x41, 'exact Is_ compound prefix'); + rejects_property('isBlock=Basic_Latin', 'lowercase is is not compound Is prefix'); + rejects_property('ISBlock=Basic_Latin', 'uppercase IS is not compound Is prefix'); + + accepts_property('Block=Greek_And_Coptic', 0x378, + 'unassigned code point remains in block'); + accepts_property('InGreek', 0x378, 'InGreek resolves to block'); + accepts_property('General_Category=Cn', 0x378, 'same code point is unassigned'); + accepts_property('Greek', 0x1F00, 'Greek resolves to script extensions'); + accepts_property('IsGreek', 0x1F00, 'IsGreek resolves to script extensions'); + + accepts_property('Block=No_Block', 0x2FE0, 'gap has default No_Block value'); + accepts_property('Blk=NB', 0x2FE0, 'NB is official No_Block alias'); + accepts_property('InNB', 0x2FE0, 'InNB shortcut'); + accepts_property('Block=Arabic_Presentation_Forms_A', 0xFDD0, + 'noncharacter U+FDD0 retains enclosing block'); + accepts_property('Noncharacter_Code_Point', 0xFDD0, 'U+FDD0 is a noncharacter'); + accepts_property('Block=Specials', 0xFFFE, 'U+FFFE retains Specials block'); + + accepts_property('Block=#\\ABasic_Latin\\z#', 0x41, 'anchored wildcard value'); + accepts_property('Block=#\\A(?:Basic_Latin|Greek_And_Coptic)\\z#', 0x378, + 'wildcard value alternation'); + rejects_property('Block=Basic*', 'bare glob is not a wildcard value'); + rejects_property('Block=#Old I.*#', 'star quantifier forbidden in wildcard'); + rejects_property('Block=#\\ANever_A_Block\\z#', + 'wildcard matching no value is rejected'); + rejects_property('Is_Block=#\\ABasic_Latin\\z#', + 'wildcard unavailable with Is_ prefix'); + + rejects_property('Block', 'Block needs a value'); + rejects_property('Blk', 'Blk needs a value'); + rejects_property('Blocks=Basic_Latin', 'invalid property alias'); + rejects_property('Block=Basic_Lati', 'invalid value alias'); + rejects_property('InDefinitelyNotABlock', 'unknown In property'); + + SKIP: { + skip 'local standard Perl uses Unicode ' . Unicode::UCD::UnicodeVersion(), 2 + if Unicode::UCD::UnicodeVersion() lt '17.0.0'; + accepts_property('Block=Sidetic', 0x10940, 'Unicode 17 Sidetic sentinel'); + accepts_property('Block=Tolong_Siki', 0x11DB0, + 'Unicode 17 Tolong Siki sentinel'); + } +} + +done_testing; From d7acd744278302cc1af55e60a93878ca3848c16a Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:18:00 +0200 Subject: [PATCH 35/60] docs(regex): record unified Phase 36 integration strategy Document the one-commit-per-source-PR integration branch, the excluded duplicate ancestry, the final validation gates, and merge-commit rollback policy. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 330dfa6d9b..006ed80a09 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -224,12 +224,19 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active -The published integration stack is preserved through draft PR #1038, stacked -on draft PRs #1025–#1026 and review-ready PR #1024. It includes the completed callback/runtime slices, -lossless generated Unicode fixtures, explicit `Is_*` property/value +The unified `integration/phase36-regex-parity` branch was assembled on +2026-08-18 from all 34 open Phase 36 PRs, with one squashed review-unit commit +per PR. Duplicate #1007 ancestry from #1010, duplicate #1008/#1009/#1012 +ancestry from #1016, and temporary integration merge commits were excluded. +The final stacked implementation matches PR #1039 plus the independent #1006, +#1007, and #1010 changes. Full integration validation and the unified draft PR +remain pending while unrelated resource-intensive Java builds finish. + +The unified history includes the completed callback/runtime slices, lossless +generated Unicode fixtures, explicit `Is_*` property/value normalization, fatal Joni syntax diagnostics, native GCB semantics, and the -first 524 lines of retired Java-only preprocessor code. Every published slice -has a warning-free combined `make` checkpoint. PR #1027 adds native sentence +first 524 lines of retired Java-only preprocessor code. Every source slice has +a warning-free combined `make` checkpoint. PR #1027 adds native sentence boundaries; PR #1028 adds independently validated alpha assertion aliases and native word boundaries. PR #1029 integrates corrected global zero-width `/g` progression and pinned Perl 5.44 Unicode 17.0 Age properties. PR #1030 adds @@ -702,9 +709,9 @@ is retained for now. ### Next Steps -1. Land review-ready PR #1024 and draft PRs #1025–#1038. Publish the validated - Block integration in a separate focused WIP PR, preserving the - independently reviewed data commit. +1. Run warning-free `make` on `integration/phase36-regex-parity`, push the + branch, and open one unified draft PR against `master`. Keep its 34 source + PR commits intact and use a merge commit after final validation. 2. Preserve the now-complete native Joni boundary corpus at 239,866/239,866: sentence chunk 05, line chunks 06–09, and word chunk 10 must remain exact on JVM and interpreter while property and parser work continues. @@ -726,8 +733,9 @@ is retained for now. stale Unicode limitations, add any still-missing regex features, and link each limitation to a reducer or explicit optimizer/debug exclusion. 7. Run unchanged CPAN consumers, the direct/thread release matrix, packaging - and license checks, and warning-free `make`; then rebase each focused PR and - require green Ubuntu and Windows CI. + and license checks, and require green Ubuntu and Windows CI on the unified + PR. Merge it with a merge commit so the focused history remains available + and the complete integration can be reverted with `git revert -m 1`. ### Open Questions and Blockers From 05741098b78b7331b457951bb5bc0cc9b3f0923d Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Tue, 18 Aug 2026 09:37:06 +0200 Subject: [PATCH 36/60] feat(regex): add pinned Script-property parity (#1040) Squashed integration of PR #1040 at eff028a41 after its explicit coordinator ready marker. Preserve pinned Script and Script_Extensions data, exact precedence/composite policy, and the final focused/corpus evidence. Generated with [Codex](https://openai.com/codex) Co-Authored-By: Codex --- dev/design/phase36-regex-parity.md | 66 +- .../generate_perl_unicode_script_data.pl | 376 ++++++++++ docs/reference/feature-matrix.md | 4 +- .../runtime/regex/JoniRegexPattern.java | 25 +- .../runtime/regex/PerlUnicodeScriptData.java | 647 ++++++++++++++++++ .../runtime/regex/UnicodeResolver.java | 129 ++++ .../regex/PerlUnicodeScriptDataTest.java | 139 ++++ .../regex/unicode_script_extensions_pinned.t | 130 ++++ 8 files changed, 1497 insertions(+), 19 deletions(-) create mode 100644 dev/tools/generate_perl_unicode_script_data.pl create mode 100644 src/main/java/org/perlonjava/runtime/regex/PerlUnicodeScriptData.java create mode 100644 src/test/java/org/perlonjava/runtime/regex/PerlUnicodeScriptDataTest.java create mode 100644 src/test/resources/unit/regex/unicode_script_extensions_pinned.t diff --git a/dev/design/phase36-regex-parity.md b/dev/design/phase36-regex-parity.md index 006ed80a09..2594c14fe4 100644 --- a/dev/design/phase36-regex-parity.md +++ b/dev/design/phase36-regex-parity.md @@ -165,7 +165,18 @@ allocate callback state or callback frames. suite whose regex capability policy is removed. 5. Run warning-free `make`, Joni upstream tests, packaging and license checks, and the thread release matrix. -6. Rebase each focused delivery slice onto current master. Require green Ubuntu +6. Rewrite `dev/implementation/regex.md` to describe the final as-implemented + matcher architecture and ownership boundaries, and update + `docs/design/joni-callout-fork.md` to match the shipped fork API, namespace, + packaging, callback/unwind contract, and Unicode responsibilities. Review + both documents for a clear reader path, consistent terminology, and removal + of superseded proposals or predictions. Audit the remaining regex/Joni + design documents: delete only content that is wholly redundant and retains + no useful rationale; otherwise replace historical implementation plans with + concise summaries that preserve decisions and point to the canonical + implementation and fork documents. Preserve copyright and authorship + notices in every retained or consolidated third-party description. +7. Rebase each focused delivery slice onto current master. Require green Ubuntu and Windows CI before merging and beginning the next slice. Exit criteria: all semantic gates pass, no previously passing file regresses, @@ -225,10 +236,10 @@ compatibility contract. ### Current Status: Phases 0, 2, and 4 complete; Phases 1 and 3 corpus gates active The unified `integration/phase36-regex-parity` branch was assembled on -2026-08-18 from all 34 open Phase 36 PRs, with one squashed review-unit commit +2026-08-18 from all 35 ready Phase 36 PRs, with one squashed review-unit commit per PR. Duplicate #1007 ancestry from #1010, duplicate #1008/#1009/#1012 ancestry from #1016, and temporary integration merge commits were excluded. -The final stacked implementation matches PR #1039 plus the independent #1006, +The final stacked implementation matches PR #1040 plus the independent #1006, #1007, and #1010 changes. Full integration validation and the unified draft PR remain pending while unrelated resource-intensive Java builds finish. @@ -246,10 +257,11 @@ empty-property rendering. PR #1032 integrates native numeric escapes through U+10FFFF; PR #1033 adds pinned Bidi_Class sets; PR #1034 integrates native vertical-whitespace escapes; PR #1035 adds Decomposition_Type and PR #1036 adds East_Asian_Width, PR #1037 adds Numeric_Value, and PR #1038 adds -Joining_Group. The current WIP integrates Block; independently validated -Script/Script_Extensions, break-property values, generic and specialized -binary-property data, and the first preprocessor dead-state deletion are ready -for focused integration. +Joining_Group, and PR #1039 adds Block. The current WIP integrates +Script/Script_Extensions; independently validated break-property values, +generic and specialized binary-property data, residual enumerated-property +families, and the first preprocessor dead-state deletion are ready for focused +integration. Lexical `use bytes` now compiles non-ASCII substitution patterns with a single-byte Joni encoding while preserving upgraded, byte-backed, and compiled @@ -391,6 +403,21 @@ boundary smoke remains 169/169 per backend, and chunks 01–04 gain 8,324 assertions with zero numbered regressions and exact backend identity. Current generated evidence is 369,462/407,367. +`Script`/`sc` and `Script_Extensions`/`scx` assignments now resolve all 176 +values from pinned Unicode 17 partitions and Script_Extensions overrides. +Explicit `sc` retains strict Script semantics while Perl's bare Script-value +shortcuts use Script_Extensions; the composite `Katakana_Or_Hiragana`/`Hrkt` +pseudo-value is rejected from bare and exact assignments and excluded from +wildcard unions as required by Perl. Loose aliases, `Qaac`/`Qaai`, +wildcards, exact `Is` assignment policy, precedence over Block shortcuts, and +positive or complemented properties inside ordinary character classes are +covered. The 95-assertion oracle passes system Perl, JVM, and interpreter; the +focused precedence, class-negation, and bare-scx reducers pass 7/7, 8/8, and +10/10 respectively on all three runtimes. Protected boundary smoke remains +169/169 per backend. Chunks 01–04 gain 8,140 assertions with zero numbered +regressions and exact JVM/interpreter counts, raising current generated +evidence to 377,602/407,367. + Joni now accepts Perl's top-level, scoped, combined, and negative inline `p` syntax as matcher-neutral policy. PerlOnJava publishes that policy while ordinary and substitution callbacks execute, without misclassifying escaped or @@ -664,6 +691,12 @@ is retained for now. partition with official aliases, ordered `No_Block` gaps, wildcard policy, and Script/category/binary precedence. Chunks 01–04 gain 8,324 assertions with zero numbered regressions and exact backend identity. + - [x] Generated and integrated complete pinned Unicode 17.0 Script and + Script_Extensions sets, including Perl's bare-scx policy, strict explicit + Script assignments, composite-value rejection and wildcard exclusion, + aliases, precedence, and ordinary character-class complements. The focused oracle + passes 95/95; chunks 01–04 gain 8,140 assertions with zero numbered + regressions and exact backend counts. - [x] Integrated native Perl `\v`/`\V` dispatch inside and outside character classes (`1eff1db97`, integrated as `6328935cd`). The focused oracle passes 92/92 and unchanged `reg_posixcc.t` passes 2,560/2,560 on both backends. @@ -710,14 +743,14 @@ is retained for now. ### Next Steps 1. Run warning-free `make` on `integration/phase36-regex-parity`, push the - branch, and open one unified draft PR against `master`. Keep its 34 source + branch, and open one unified draft PR against `master`. Keep its 35 source PR commits intact and use a merge commit after final validation. 2. Preserve the now-complete native Joni boundary corpus at 239,866/239,866: sentence chunk 05, line chunks 06–09, and word chunk 10 must remain exact on JVM and interpreter while property and parser work continues. -3. Integrate the independently generated Script/Script_Extensions and - break-property value slices, then the generic - and specialized binary-property families currently advancing in parallel. +3. Integrate the independently generated break-property value slice, then the + generic and specialized binary-property families and residual enumerated + families currently advancing in parallel. Preserve pinned Perl 5.44 acceptance and rejection semantics rather than inheriting host ICU breadth. Keep native `\v`/`\V` exact at 2,560/2,560 in `reg_posixcc.t`. @@ -729,9 +762,14 @@ is retained for now. Move matcher semantics into Joni, retain only source-policy scanning, delete Java-only rewrites and compiled-pattern variants, and remove the temporary Java backend selector after the performance gate passes. -6. Reconcile `docs/reference/feature-matrix.md` with the final corpus: replace - stale Unicode limitations, add any still-missing regex features, and link - each limitation to a reducer or explicit optimizer/debug exclusion. +6. Reconcile `docs/reference/feature-matrix.md` with the final corpus; update + `dev/implementation/regex.md` and `docs/design/joni-callout-fork.md` to the + as-implemented architecture and review both for clarity and structure. + Audit redundant regex/Joni documents, deleting only wholly redundant text + and summarizing historically useful rationale with links to the canonical + documents. Replace stale Unicode limitations, add any still-missing regex + features, and link each limitation to a reducer or explicit optimizer/debug + exclusion. 7. Run unchanged CPAN consumers, the direct/thread release matrix, packaging and license checks, and require green Ubuntu and Windows CI on the unified PR. Merge it with a merge commit so the focused history remains available diff --git a/dev/tools/generate_perl_unicode_script_data.pl b/dev/tools/generate_perl_unicode_script_data.pl new file mode 100644 index 0000000000..d3286951d2 --- /dev/null +++ b/dev/tools/generate_perl_unicode_script_data.pl @@ -0,0 +1,376 @@ +#!/usr/bin/env perl +use strict; +use warnings; +use Digest::SHA qw(sha256_hex); +use File::Spec; +use FindBin; + +my $expected_version = '17.0.0'; +my $root = File::Spec->catdir($FindBin::Bin, '..', '..'); +my $unicore = File::Spec->catdir($root, 'perl5', 'lib', 'unicore'); +my @sources = ( + { + name => 'Scripts-17.0.0.txt', + path => File::Spec->catfile($unicore, 'Scripts.txt'), + hash => '9f5e50d3abaee7d6ce09480f325c706f485ae3240912527e651954d2d6b035bf', + version => qr/^# Scripts-\Q$expected_version\E\.txt$/m, + }, + { + name => 'ScriptExtensions-17.0.0.txt', + path => File::Spec->catfile($unicore, 'ScriptExtensions.txt'), + hash => 'ec2107e58825a1586acee8e0911ce18260394ac8b87e535ca325f1ccbeb06bc6', + version => qr/^# ScriptExtensions-\Q$expected_version\E\.txt$/m, + }, + { + name => 'PropertyValueAliases-17.0.0.txt', + path => File::Spec->catfile($unicore, 'PropValueAliases.txt'), + hash => '670d2bebb48649c04fabfbf033308073dcff47946324a8033237254c048b3b01', + version => qr/^# PropertyValueAliases-\Q$expected_version\E\.txt$/m, + }, + { + name => 'PropertyAliases-17.0.0.txt', + path => File::Spec->catfile($unicore, 'PropertyAliases.txt'), + hash => '4441f573caf952ffece1d7c892e7715bd7136dfc26f96eb6f268bf1e474715fb', + version => qr/^# PropertyAliases-\Q$expected_version\E\.txt$/m, + }, +); + +sub read_source { + my ($source) = @_; + open my $input, '<:raw', $source->{path} + or die "Cannot read $source->{path}: $!\n"; + local $/; + my $text = <$input>; + close $input or die "Cannot close $source->{path}: $!\n"; + my $actual_hash = sha256_hex($text); + die "$source->{path} SHA-256 mismatch: expected $source->{hash}, found $actual_hash\n" + unless $actual_hash eq $source->{hash}; + die "$source->{path} is not pinned Unicode $expected_version data\n" + unless $text =~ $source->{version}; + $source->{text} = $text; +} + +sub trim { + my ($text) = @_; + $text =~ s/^\s+|\s+$//g; + return $text; +} + +sub loose_name { + my ($name) = @_; + $name = lc $name; + $name =~ s/[\s_-]+//g; + return $name; +} + +sub parse_range { + my ($text) = @_; + my ($start, $end) = split /\.\./, $text; + return (hex($start), hex(defined $end ? $end : $start)); +} + +read_source($_) for @sources; + +my $version_path = File::Spec->catfile($unicore, 'version'); +open my $version_input, '<', $version_path or die "Cannot read $version_path: $!\n"; +chomp(my $unicode_version = <$version_input>); +close $version_input or die "Cannot close $version_path: $!\n"; +die "Expected Unicode $expected_version, found $unicode_version\n" + unless $unicode_version eq $expected_version; + +my (@short_values, @long_values, %alias_index, %wildcard_value_index); +for my $line (split /\n/, $sources[2]{text}) { + next if $line =~ /^\s*#/; + $line =~ s/#.*$//; + my @fields = map { trim($_) } split /;/, $line; + next unless @fields >= 3 && $fields[0] eq 'sc'; + my $index = scalar @short_values; + push @short_values, $fields[1]; + push @long_values, $fields[2]; + for my $alias (@fields[1 .. $#fields]) { + next unless length $alias; + my $loose = loose_name($alias); + die "Script alias collision for '$alias'\n" + if exists $alias_index{$loose} && $alias_index{$loose} != $index; + $alias_index{$loose} = $index; + die "Script wildcard alias collision for '$alias'\n" + if exists $wildcard_value_index{$alias} + && $wildcard_value_index{$alias} != $index; + $wildcard_value_index{$alias} = $index; + } +} +die "Expected 176 Script values, found " . scalar(@short_values) . "\n" + unless @short_values == 176; + +my (%script_property_aliases, %script_extensions_property_aliases); +for my $line (split /\n/, $sources[3]{text}) { + next if $line =~ /^\s*#/; + $line =~ s/#.*$//; + my @fields = map { trim($_) } split /;/, $line; + next unless @fields >= 2; + if ($fields[0] eq 'sc' || $fields[1] eq 'Script') { + $script_property_aliases{loose_name($_)} = 1 for grep { length } @fields; + } + if ($fields[0] eq 'scx' || $fields[1] eq 'Script_Extensions') { + $script_extensions_property_aliases{loose_name($_)} = 1 + for grep { length } @fields; + } +} +die "Pinned property aliases do not define sc and Script\n" + unless $script_property_aliases{sc} && $script_property_aliases{script}; +die "Pinned property aliases do not define scx and Script_Extensions\n" + unless $script_extensions_property_aliases{scx} + && $script_extensions_property_aliases{scriptextensions}; + +my (@script_missing, @script_explicit); +for my $line (split /\n/, $sources[0]{text}) { + if ($line =~ /^#\s*\@missing:\s*([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z_]+)/) { + my ($range, $value) = ($1, $2); + my ($start, $end) = parse_range($range); + my $index = $alias_index{loose_name($value)}; + die "Unknown \@missing Script value '$value'\n" unless defined $index; + push @script_missing, [$start, $end, $index]; + next; + } + next unless $line =~ /^([0-9A-F]+(?:\.\.[0-9A-F]+)?)\s*;\s*([A-Za-z_]+)/; + my ($range, $value) = ($1, $2); + my ($start, $end) = parse_range($range); + my $index = $alias_index{loose_name($value)}; + die "Unknown explicit Script value '$value'\n" unless defined $index; + push @script_explicit, [$start, $end, $index]; +} +die "Script data has no ranges or defaults\n" + unless @script_explicit && @script_missing; + +my $unknown_index = $alias_index{loose_name('Unknown')}; +my @script_code_value = ($unknown_index) x 0x110000; +for my $range (@script_missing) { + my ($code, $end, $index) = @$range; + $script_code_value[$code++] = $index while $code <= $end; +} +for my $range (@script_explicit) { + my ($code, $end, $index) = @$range; + $script_code_value[$code++] = $index while $code <= $end; +} + +my @script_ranges; +my ($range_start, $range_value) = (0, $script_code_value[0]); +for my $code (1 .. 0x10ffff) { + next if $script_code_value[$code] == $range_value; + push @script_ranges, [$range_start, $code - 1, $range_value]; + ($range_start, $range_value) = ($code, $script_code_value[$code]); +} +push @script_ranges, [$range_start, 0x10ffff, $range_value]; + +my (@script_extensions_ranges, @script_extensions_values); +my @script_extensions_explicit = (0) x 0x110000; +my $saw_script_fallback = 0; +for my $line (split /\n/, $sources[1]{text}) { + if ($line =~ /^#\s*\@missing:\s*0000\.\.10FFFF\s*;\s*