[feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer - #64667
[feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer#64667nishant94 wants to merge 28 commits into
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
@nishant94 have you tried icu analyzer? because I think icu could handle many different languages. |
@yiguolei The ICU Analyzer is not good as the Kuromoji. There is huge difference between icu and kuromoji when it comes to morphology of the Japanese words. So I think it worth it adding this new parser. |
|
Is the code under |
This is original code but it is modeled on Apache Lucene's kuromoji. |
FE UT Coverage ReportIncrement line coverage |
389fcfb to
b79db3c
Compare
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
db0ee69 to
06b4ef6
Compare
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
…wn words - Implemented functionality in the Kuromoji Viterbi segmenter to decompose unknown (out-of-vocabulary) words into per-character unigrams when in extended mode, aligning with Lucene's JapaneseTokenizer behavior. - Added unit tests to validate the correct segmentation of unknown words in both normal and extended modes, ensuring expected outputs for various input scenarios.
- Modified error messages to include 'kuromoji' parser in the parser mode validation. - Enhanced tests for the Japanese analyzer to assert expected tokenization results.
- Introduced a new configuration option `enable_kuromoji_analyzer` to toggle the Kuromoji analyzer functionality. - Updated unit tests to validate the behavior of the Kuromoji analyzer when enabled and disabled. - Modified tests to enable the Kuromoji analyzer for specific test cases.
- Updated the namespace for Kuromoji components from `doris::segment_v2::kuromoji` to `doris::segment_v2::inverted_index::kuromoji` across multiple files for better organization and clarity.
- Updated the CMake configuration to ensure the required Kuromoji dictionary files are present at build time, failing the build if any are missing. - Modified the KuromojiAnalyzer and KuromojiTokenizer to throw exceptions when the dictionary is not loaded, preventing silent fallbacks to per-codepoint tokenization. - Improved error handling and validation in the dictionary loading process to ensure robust operation. - Updated unit tests to validate the new behavior, ensuring that missing dictionaries trigger appropriate errors.
- Replaced the `ending_at` vector with `end_head` and `end_next` for better memory management and performance during node processing. - Updated node addition and traversal logic to utilize the new data structures, enhancing the segmenter's efficiency in handling word segmentation.
- Changed the values in the `unk.per_category[CAT_DEFAULT]` entry from `{5, 5, 4769, "unk-default"}` to `{2, 2, 4769, "unk-default"}` to correct the test setup.
- Modified CMake configuration to conditionally include the Kuromoji dictionary files only for non-test builds (MAKE_TEST=ON). - Adjusted the custom target for generating the Kuromoji dictionary to reflect the new conditional behavior, ensuring it remains a manual target during unit-test builds.
- Added checks for empty trie and out-of-range category mappings in the Kuromoji dictionary. - Updated tests
- Added logic to return the Kuromoji search mode based on the analyzer property. - Updated unit tests accordingly.
- This enhancement ensures that the necessary Kuromoji dictionary source is available for builds, improving the setup process for users.
- Updated test cases in `test_japanese_analyzer.groovy` to use query-time (qt_) assertions for better readability and maintainability.
- Introduced a penalty calculation mechanism for segmenting words based on the analyzer mode, improving the accuracy of word segmentation. - Added caching for the current same-category run's byte end to optimize processing of grouped unknown words, reducing unnecessary rescans.
- Moved mecab-ipadic staging into a conditional block based on the BUILD_BE flag.
- Refactored the logic for setting the Kuromoji analyzer mode to ensure it defaults to the search mode when the parser mode is empty or set to coarse granularity.
- Introduced sorting of CSV file paths before processing to ensure consistent order. - Enhanced error handling during lexicon parsing, including detailed error messages for failed parses. - Added sorting of BuilderWord entries by cost and IDs.
- Implemented a new utility function to decode the first UTF-8 code point from a given string view, enhancing character processing capabilities. - Updated the KuromojiTokenizer to skip unknown tokens based on their character category, improving tokenization accuracy.
- Introduced a new executable `kuromoji_build_dict` to compile the UTF-8 mecab-ipadic source into binary files. - Added custom commands and targets to generate the Kuromoji dictionary, ensuring it is built as part of the overall project unless in test mode.
- Updated the CMake configuration to prepend the Kuromoji JVM library directory to the DYLD_LIBRARY_PATH and LD_LIBRARY_PATH environment variables during the dictionary generation process.
- Updated CMake configuration to set DYLD_LIBRARY_PATH and LD_LIBRARY_PATH for better library management during dictionary generation. - Improved README documentation for manual dictionary regeneration steps. - Refactored the Kuromoji dictionary loading logic to cache only successful loads, enhancing error handling. - Added assertions in the Japanese analyzer tests to ensure proper configuration and cleanup of the Kuromoji analyzer state.
- Replaced `qt_` queries with direct SQL calls and assertions to validate results in the Japanese analyzer test suite.
- Updated the build script to allow compilation of the Kuromoji dictionary when the COMPILE_BENCH flag is set. - Modified the dictionary building logic to write temporary files before renaming them atomically, improving reliability during the build process. - Ensured that the mecab-ipadic staging in the third-party build script uses a temporary directory for safer file operations.
c6e4849 to
5bda577
Compare
|
run buildall |
FE Regression Coverage ReportIncrement line coverage |
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Request changes: three distinct correctness/build-isolation issues remain; the inline comments give concrete triggers and fix/test directions.
Critical checkpoint conclusions:
- Goal/proof: the PR adds an opt-in Kuromoji/IPADIC analyzer across DDL, index writes, MATCH-family queries, and TOKENIZE. The main Japanese morphology path is wired and tested, but Unicode lowercasing, malformed-byte handling, and build-output provenance are not correct.
- Scope/focus: the 51-file change is broad but cohesive across FE metadata, BE runtime, dictionary compilation/packaging, and tests. No additional review focus was supplied, so the complete scope was reviewed.
- Concurrency: runtime dictionary publication is mutex-protected and successful mappings are immutable; analyzer/tokenizer state is instance-local. Independent build trees can race or reuse the shared generated dictionary directory (inline).
- Lifecycle/static initialization: factory, load, cache, mmap, and error lifecycles were traced. No new static-initialization issue was found; existing failure-cache and mmap-accounting concerns already have live threads.
- Configuration: the mutable default-off gate reaches writer, query, and TOKENIZE construction paths. No distinct gate-refresh issue was found beyond existing threads.
- Compatibility: no wire enum or storage-format version is added, but string metadata crosses FE/BE and rolling upgrades. Existing custom-name, default-mode, analyzer-validation, and mixed-version concerns were not duplicated.
- Parallel paths: writer, indexed and slow MATCH/phrase paths, and TOKENIZE were traced. The lowercasing and malformed-byte findings affect the shared analyzer path.
- Conditional checks: mode, property, gate, parser, loader, and dictionary validation branches were reviewed; no additional distinct conditional-check defect remains.
- Test coverage: synthetic BE, FE validation, real-dictionary, and regression paths were inspected. Unicode case, invalid-byte, and two-build-tree provenance coverage is missing as described inline; other missing coverage already has live comments.
- Test results: the changed expected errors and query assertions were inspected. Existing deterministic-output and config-restoration concerns were suppressed as duplicates. No build or test was run in this review-only environment.
- Observability: load and validation failures log or propagate errors; no new distinct logging/metric gap was found. Existing mmap accounting feedback was not repeated.
- Transactions/persistence: catalog property persistence/replay was traced and no transaction/EditLog-specific defect was found.
- Data writes: index-term generation and dictionary publication/package installation were reviewed. Wrong shared-build provenance can package incompatible terms (inline); existing multi-file atomic-publication feedback was not duplicated.
- FE-BE variables: analyzer/parser/mode properties use existing metadata and thrift maps; DDL, replay, writer, indexed/slow query, and TOKENIZE send/receive paths were checked.
- Performance: lattice/OOV complexity, copies, cache locking, and mmap cost were reviewed. Known performance concerns already have live threads; no new distinct issue was found.
- Other issues: licensing, vendoring, formatting, registration, and documentation changes were inspected; no further distinct issue remains.
Review completion: converged in round 3. All final normal and risk-focused passes returned NO_NEW_VALUABLE_FINDINGS after validating the three inline findings.
|
|
||
| set(KUROMOJI_IPADIC_SRC "${THIRDPARTY_DIR}/share/mecab-ipadic-2.7.0-20250920" | ||
| CACHE PATH "UTF-8 mecab-ipadic source directory used to generate the kuromoji dictionary") | ||
| set(KUROMOJI_DICT_OUT "${BASE_DIR}/dict/kuromoji") |
There was a problem hiding this comment.
[Major] Keep generated dictionary outputs scoped to the current build tree. KUROMOJI_IPADIC_SRC is a per-tree cache variable and the README documents overriding it, but KUROMOJI_DICT_OUT is this shared source-tree directory and the install rule copies it. If tree B successfully generates from a custom source, its newer files can make tree A's default-source edge look up to date, so A silently packages B's dictionary; concurrent trees also race the same temporary paths. Generate and install from a build/config-specific directory with a provenance/completion stamp, and cover two build trees configured with different sources.
| pos1 == "\xE3\x83\x95\xE3\x82\xA3\xE3\x83\xA9\xE3\x83\xBC"; // フィラー (filler) | ||
| } | ||
|
|
||
| void ascii_lower(std::string& s) { |
There was a problem hiding this comment.
[Major] Apply the full Unicode lower-case contract here. When lower_case=true, writer, indexed/slow MATCH, and TOKENIZE all reach this tokenizer, but ascii_lower() changes only A-Z. For example, an indexed ΜΈΓΑ remains uppercase while the query μέγα emits a different term, even though Doris's shared LowerCaseFilter and its tests define Unicode-aware folding (ÜBER ΜΈΓΑ -> über μέγα). Reuse the existing ICU-based implementation (while preserving lower_case=false), and add non-ASCII TOKENIZE plus persisted index/query coverage.
|
|
||
| // Decode one UTF-8 code point at text[pos]. Invalid/truncated -> single byte. | ||
| DecodedCp decode_utf8(std::string_view text, std::size_t pos) { | ||
| auto b0 = static_cast<unsigned char>(text[pos]); |
There was a problem hiding this comment.
[Major] Honor the documented one-byte fallback for malformed UTF-8. For bytes C3 41, this branch only checks the lead byte and remaining length, treats 41 as a continuation, and consumes the valid A inside one two-byte morpheme. The tokenizer later copies C3 41 and lowercases it to invalid C3 61, so TOKENIZE can return malformed bytes and indexing loses the separately searchable A. Doris STRING values can carry such input via unhex. Use one strict shared decoder for Viterbi/category/width normalization (validating continuations, shortest form, surrogates, and U+10FFFF), choose an explicit invalid-input policy, and test short mixed-invalid inputs end to end.
7 participants
What problem does this PR solve?
Issue Number: #64646
Related PR: None
Problem Summary:
Doris has no Japanese-aware tokenizer for the inverted index. Japanese text has no spaces between words, so the existing parsers can't segment it and
MATCH/MATCH_PHRASEon Japanese columns end up with poor recall and precision.This PR adds a built-in
kuromojiparser for Japanese, in the same style as the existing Chinese IK analyzer. It's opt-in per column:After indexing, MATCH, MATCH_PHRASE and TOKENIZE() run against the segmented Japanese terms.
How it works:
be/src/storage/index/inverted/analyzer/kuromoji/, so there's no JVM on the indexing path. KuromojiAnalyzer / KuromojiTokenizer mirror the IK analyzer/tokenizer, with a Viterbi cost-model segmenter over the IPADIC connection-cost matrix.Dictionary source is mecab-ipadic-2.7.0-20070801 (NAIST-2003 license, the same lexicon Lucene kuromoji uses).
Release note
Support Japanese text tokenization in the inverted index via a new kuromoji parser (
PROPERTIES("parser"="kuromoji")), withsearch/normal/extendedmodes.Check List (For Author)
parser="kuromoji".Check List (For Reviewer who merge this PR)