Implement German compound word tokenizer and fix head word capitalization

[nciric]

This PR implements the German compound word tokenizer (DeTokenizer) and integrates it into the inflection engine. It also resolves capitalization lookup issues for lowercased head-word segments resulting from compound splitting.

Key Changes

1. German Tokenizer Infrastructure & Configuration (DeTokenizer)

• Implemented DeTokenizer and DeDictionaryTokenizerConfig registered in TokenizerFactory.cpp.
• Configured German to use the new tokenizer and compiled binary dictionary (/de/tokenizer.tokd) in config_de.properties.
• Configured German Fugenelemente (s, es, e, n, en, er, ens) in DeDictionaryTokenizerConfig.

2. German Head-Word Capitalization Lookup

• Lowercased compound head-word segments (e.g. frau from Ehefrau) are checked against capitalized noun entries (Frau) in dictionary.getCombinedBinaryType to resolve noun POS and feminine gender morphology, restoring lower-case formatting on inflectedHeadWord after inflection synthesis.

3. German Tokenizer Dictionary

• Added tokenizer.dictionary for German containing word frequencies and flags (isNoCompound, isNoAtomic, isSegment).

Verification

• Added unit tests in TokenizerTest.cpp covering German compound splitting (Hauptstraße, Waldweg, Schlossgasse, Marktplatz, Kaiserring, Sonnenallee, Arbeitszimmer, Kindergarten).
• Executed make check - all 264 test cases passed (290,502 assertions).

[nciric]

Technical Investigation: Case Sensitivity in DictionaryMetaData Lookup

Following review discussion regarding case sensitivity during dictionary lookup, we investigated how DictionaryMetaData::getCombinedBinaryType and underlying Trie lookups behave in the engine:

1. Trie Byte Matching

Dictionary entries in the binary dictionary (e.g. Frau, Weg, Strasse) are stored in a marisa::Trie (MarisaTrie.hpp), which performs exact byte-sequence lookups.

2. One-Way Lowercasing Fallback in getCombinedBinaryType

In DictionaryMetaData::getCombinedBinaryType (DictionaryMetaData.cpp):

int64_t* DictionaryMetaData::getCombinedBinaryType(int64_t* result, std::u16string_view word) const
{
    *npc(result) = 0;
    auto combinedType = dictionary->getWordType(word);
    if (!combinedType) {
        ::std::u16string normalized;
        transform(&normalized, word, dictionary->getLocale()); // Lowercases word
        if (normalized != word) {
            combinedType = dictionary->getWordType(normalized);
        }
    }
    if (!combinedType) {
        return nullptr;
    }
    *npc(result) = *combinedType;
    return result;
}

• Uppercase/Capitalized inputs (Frau / FRAU): First checks exact match Frau, then falls back to lowercasing frau.
• Lower-case inputs (frau): Searches frau (fails, as entry is stored as Frau), then transform lowercases frau -> frau (normalized == word), returning nullptr.

Conclusion for German Compound Tokenization

When a compound noun like Ehefrau is split into Ehe + frau, the head segment frau is lowercased. Because getCombinedBinaryType does not perform capitalization fallback for lowercased inputs, capitalizing frau -> Frau before getCombinedBinaryType is necessary for frau to resolve noun properties (dictionaryNoun | dictionaryFeminine) from capitalized dictionary entries.

[grhoten]

• Lower-case inputs (frau): Searches frau (fails, as entry is stored as Frau), then transform lowercases frau -> frau (normalized == word), returning nullptr.

This is a bad assumption. Frau is first, and it gets 2 entries in buildDictionary. One is exact casing, and the other is lower cased. Then frau is encountered later. It's an exact match, and overwrites the grammemes from Frau as frau, but I believe that it pushes the possible inflection to the end of the list of inflections.

It does not return null in this scenario.

I remain skeptical on the letter casing of this pull request.

[nciric]

Correction + deeper investigation on dictionary lookup casing

@george you're right, and my earlier comment was wrong on the mechanism: getCombinedBinaryType("frau") does not return nullptr. Thanks for pushing on this. I verified empirically against the built German dictionary (temporary diagnostic test, since reverted):

=== frau (lowercase) result non-null: 1
=== frau (lowercase) properties: indefinite pronoun
=== Frau (uppercase) result non-null: 1
=== Frau (uppercase) properties: accusative dative feminine genitive nominative noun singular
=== frau has noun bit: 0
=== Frau has noun bit: 1

The lookup succeeds, but resolves to the wrong entry. dictionary_de.lst has two real entries:

Frau: singular accusative dative genitive nominative feminine noun inflection=1
frau: indefinite pronoun inflection=13b

frau is a genuine German word (indefinite pronoun), not a casing artifact. In buildDictionary (Dictionary.cpp::extractPartsOfSpeech):

• Frau is processed first: wordsToTypes["Frau"] = {noun,feminine,…} (operator[]), and since the normalized form differs, emplace("frau", {noun,feminine,…}) inserts a lowercase copy.
• frau is processed later: wordsToTypes["frau"] = {indefinite,pronoun} via operator[], which overwrites the normalized copy. (The emplace at line 195 can't reinstate it — the key already exists.)

Net: frau -> {indefinite,pronoun} (noun bit 0, paradigm 13b); Frau -> {noun,feminine,…} (paradigm 1). So when Ehefrau decompounds into Ehe + frau, the head-word lookup returns the pronoun homonym, (type & dictionaryNoun) == 0, and it inflects with the pronoun paradigm. The inflection test einer Ehefrau -> eine Ehefrau fails without the capitalization fix.

Could we instead fix buildDictionary so the lowercase form keeps the noun grammemes?

I tested this directly: changed both writes to merge grammemes (wordsToTypes[phrase] |= combinedType; and wordsToTypes[normalizedKey] |= combinedType;), rebuilt, and ran the full suite. It regresses other languages:

Locale	Input	Expected	Got
de	decken {accusative, singular}	decke	decken
en	Jones {genitive}	Jones's	Jones'

Folding a capitalized word's grammemes onto its lowercase homonym pollutes the POS/number/gender bits that downstream inflection rules rely on — which is exactly what the existing comment warns against ("Don't override previous value because this isn't an exact value").

And even ignoring the regressions, it's the wrong layer: the inflection lookup is separate and case-sensitive. DictionaryLookupInflector resolves patterns via getInflectionPatternsForWord(word) keyed by the exact string, retrying only with a lowercased variant (never capitalized), and InflectionDictionary does no case-folding. So frau can only ever resolve to paradigm 13b; the noun paradigm 1 lives exclusively under Frau. Merging grammemes would hand frau the noun bits but still leave it inflecting with the pronoun paradigm.

Conclusion: capitalizing the compound head word to Frau in DeGrammarSynthesizer is the correct, targeted layer — it fixes the lookup key for both the grammeme metadata and the inflection paradigm, scoped to German compounds rather than globally reshaping the dictionary data model. Not a null-lookup problem (my mistake), but a lowercase-homonym-shadowing problem.

We can discuss this more in the meeting.