diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d1dd95..a4b88b6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -73,9 +73,14 @@ releases may contain breaking changes. ranges/range-elements within their own document — matching the C# `Validator`'s document-wide guid scan), dangling-ref, range-parent, undefined-range-value (every grammatical-info and range-keyed trait - reachable from an entry, however nested \+ NFC-normalized), + reachable from an entry, however nested), normalization-mismatch, duplicate-form-lang, missing-media, dangling-ranges-href, and (opt-in via - `require_ids`) missing-id. + `require_ids`) missing-id. Names are resolved against range and + range-element ids under NFC normalization: FLEx can write an id in NFD while + the trait name, value, or `parent` referring to it is NFC. What matched only + after normalizing is reported as a normalization-mismatch warning, once per + id, so the encoding split stays visible to consumers that compare raw + strings. - Canonical sort: `Lexicon.sort()` / `RangesFile.sort()` (entries by case-folded guid/id, ranges/range-elements by id, field definitions by tag; informed by the C# LiftSorter, locale-independent) and diff --git a/docs/en/csharp-differences.md b/docs/en/csharp-differences.md index f6158d0..749e7f3 100644 --- a/docs/en/csharp-differences.md +++ b/docs/en/csharp-differences.md @@ -31,7 +31,7 @@ The C# `Validator` runs one RELAX NG pass and reports the first errors as string - **Invalid URIs are warnings, not errors.** The C# RELAX NG engine never enforced the `anyURI` datatype, so FieldWorks (FLEx) has been writing `file://C:/...` hrefs into real lexicons for years. Rejecting those files would flag virtually every FLEx export. - **Schematron rules are enforced** (as semantic checks): duplicate form languages and similar co-constraints in the LIFT grammar were silently ignored by both C# and raw lxml validation. -- **Cross-file comparisons are Unicode-normalized**, because FLEx writes the `.lift` in NFC and the companion `.lift-ranges` in NFD. +- **Range id comparisons are Unicode-normalized** (NFC), because FLEx's own export is not internally consistent: it normalizes to NFC on the way out, but a few writes bypass that step and emit the NFD it holds in memory. A `grammatical-info` or `lexical-relation` range-element `id` can therefore be NFD while its labels, the `parent` attribute on that same element, and the `.lift` value referring to it are all NFC. References that resolve only after normalizing are reported as `normalization-mismatch` warnings — a check with no C# counterpart — so the encoding split stays visible to anyone whose own comparisons are exact. sil-lift also validates the `.lift-ranges` companions of a loaded lexicon against a schema for standalone ranges documents (vendored from `lift-standard` alongside the base LIFT grammar) — every tracked external ranges file is checked whenever the `.lift` is validated — with no such schema (or check) in the C# world. (There is no entry point for validating a `.lift-ranges` file on its own, detached from a `.lift`.) diff --git a/docs/en/guides/lift-export-interop.md b/docs/en/guides/lift-export-interop.md index 1777e35..046a8e8 100644 --- a/docs/en/guides/lift-export-interop.md +++ b/docs/en/guides/lift-export-interop.md @@ -92,7 +92,7 @@ The companion carries each range's full definition. Values are `` ``` -An entry then refers to a value by id: a sense's part of speech is ``, and a semantic domain is ``. `sil-lift validate` warns (`undefined-range-value`) when a value isn't defined in its range and errors (`range-parent`) when a `parent` isn't a sibling id — so emit the ranges your data actually uses. See also [Ranges and media](folder-media.md). +An entry then refers to a value by id: a sense's part of speech is ``, and a semantic domain is ``. `sil-lift validate` warns (`undefined-range-value`) when a value isn't defined in its range and errors (`range-parent`) when a `parent` isn't a sibling id — so emit the ranges your data actually uses. Those comparisons are NFC-normalized, so an id and the value or `parent` referring to it may differ in Unicode normalization — that difference is a `normalization-mismatch` warning rather than an error, but write one consistent normalization if you can: consumers that compare raw strings will not resolve those references. See also [Ranges and media](folder-media.md). If you build the export in Python, `Lexicon.add_ranges_file()`, `RangesFile.add_range()`, and `Range.add_element()` construct the companion and add the header references for you; `open_writer(..., ranges=...)` does the same on the streaming path. diff --git a/docs/en/guides/validate.md b/docs/en/guides/validate.md index c8545e7..1fb66f1 100644 --- a/docs/en/guides/validate.md +++ b/docs/en/guides/validate.md @@ -18,13 +18,13 @@ lex = sil_lift.load("dictionary.lift") problems = list(lex.iter_problems()) ``` -Each `Problem` carries `level` (`"error"`/`"warning"`), a stable `code`, `message`, and an address: `file`, `entry_id`, `guid`, `line`. +Each `Problem` carries `level` (`"error"`/`"warning"`), a stable `code`, `message`, and an address: `file`, plus `entry_id`, `guid`, and `line` when the finding belongs to one entry. Findings about a document rather than an entry — `range-parent`, `normalization-mismatch`, `dangling-ranges-href` — carry the file alone (for a finding about a range, the `.lift-ranges` companion that defines it), and the unset fields are `None` (`null` in `--format json`, where every key is always present). ## The layers 1. **RELAX NG** against the LIFT 0.13 grammar (vendored from lift-standard — a byte-identical copy committed into this package). 2. **Ranges schema** — this project's `lift-ranges-0.13.rng` — over every tracked `.lift-ranges` companion. -3. **Semantic checks** the grammar cannot express: `duplicate-guid`, `dangling-ref`, `range-parent`, `undefined-range-value`, `duplicate-form-lang`, `missing-media`. +3. **Semantic checks** the grammar cannot express: `duplicate-guid`, `dangling-ref`, `range-parent`, `undefined-range-value`, `normalization-mismatch`, `duplicate-form-lang`, `missing-media`. ## Real-world FieldWorks (FLEx) output @@ -32,5 +32,8 @@ FieldWorks systematically writes some content that strict tooling rejects. Here - `file://C:/...` hrefs (invalid URIs) are reported as **warnings** (`uri-not-rfc`), not schema errors — the C# validator never rejected them. - Legally interleaved children (e.g. `field, note, field, note` in a sense) are **not** flagged, working around a false positive in libxml2. -- Range values are compared under Unicode NFC normalization — FLEx writes the `.lift` in NFC but the `.lift-ranges` in NFD within the same export. - FLEx's `trait`/`field` extensions inside `range-element` **are** reported (schema errors against the ranges schema): they are genuine spec deviations. +- Names are resolved against range and range-element `id`s under Unicode **NFC normalization** — `parent` links, range values, and the `trait` name or header `range` id that keys a range. FLEx normalizes to NFC on export, but a few writes bypass that step, so a range-element `id` can be NFD while its labels, its own `parent`, and the `.lift` values naming it are NFC. + - Compared exactly, a sound export looks broken — and a range whose `id` is spelled the other way goes unchecked entirely, since a trait name that reaches no range is silently accepted. + - A name that matched only after normalizing is reported as a `normalization-mismatch` **warning**, once per id however many references differ, addressed to the file that defines it. The data is sound, but a consumer comparing raw strings — a Send/Receive merge, say — will not resolve those references. + - The ids are never rewritten: the file keeps the spellings it came with. diff --git a/src/sil_lift/_validate.py b/src/sil_lift/_validate.py index 39e0b25..1c9e4ce 100644 --- a/src/sil_lift/_validate.py +++ b/src/sil_lift/_validate.py @@ -20,14 +20,24 @@ left untouched. 2. The vendored ``lift-ranges-0.13.rng`` over each tracked ``.lift-ranges`` companion. -3. Semantic checks the grammar cannot express: duplicate GUIDs (entries, and - ranges/range-elements within their own document), dangling ``relation/@ref`` - and ``variant/@ref``, ``range-element/@parent`` integrity, undefined range - values (every grammatical-info and range-keyed trait anywhere in the entry, - however deeply nested), duplicate form languages (the RNG's Schematron - rule, which lxml ignores), missing media files, header ``range/@href`` - references that resolve to no companion, and — opt-in — entries/senses - missing a stable id. +3. Semantic checks that the grammar cannot express, one ``Problem`` code each: + + - ``duplicate-guid`` — a guid reused, among entries or among the + ranges/range-elements of one document. + - ``dangling-ref`` — a ``relation/@ref`` or ``variant/@ref`` matching no + entry or sense. + - ``range-parent`` — a ``range-element/@parent`` no sibling id defines. + - ``undefined-range-value`` — a grammatical-info or range-keyed trait value + the range does not list, checked however deeply nested in the entry. + - ``normalization-mismatch`` — a name that reaches the range or + range-element id it refers to only under Unicode NFC normalization, + because FLEx writes some ids in NFD and the references to them in NFC. + - ``duplicate-form-lang`` — two forms in one multitext sharing a language + (the RNG's Schematron rule, which lxml ignores). + - ``missing-media`` — a referenced audio or picture file not on disk. + - ``dangling-ranges-href`` — a header ``range/@href`` resolving to no + companion file. + - ``missing-id`` — opt-in: an entry without a guid, a sense without an id. """ from __future__ import annotations @@ -45,7 +55,7 @@ if TYPE_CHECKING: import os - from collections.abc import Iterator + from collections.abc import Collection, Iterator from ._header import Range from ._model import Entry, Sense @@ -62,8 +72,8 @@ class Problem: level: Literal["error", "warning"] code: str # "schema", "duplicate-guid", "dangling-ref", "range-parent", - # "undefined-range-value", "duplicate-form-lang", "missing-media", - # "uri-not-rfc", "dangling-ranges-href", "missing-id" + # "undefined-range-value", "normalization-mismatch", "duplicate-form-lang", + # "missing-media", "uri-not-rfc", "dangling-ranges-href", "missing-id" message: str file: Path | None = None entry_id: str | None = None @@ -417,19 +427,85 @@ def _main_doc_guids() -> Iterator[tuple[str, str, str | None, int | None]]: line=at(index), ) - # Range integrity over the merged view (inline + companions). + # FLEx writes some ids in NFD but every reference to them in NFC, so + # resolving a name to an id has to compare both forms. + def nfc(value: str) -> str: + return unicodedata.normalize("NFC", value) + + # Names that resolved only after normalizing, keyed by what they resolved + # to: (range id, what the id is, id as written) -> one differing reference. + # Warning per id rather than per reference keeps a real export to 6 + # findings instead of 82. + mismatched: dict[tuple[str, str, str], str] = {} + + def id_lookup(ids: Collection[str]) -> dict[str, str]: + """What each spelling of these ids resolves to: the ids as written, + keyed by both themselves and their NFC form. An id present as written + maps to itself, so a collection holding two normalizations of one name + resolves an exact reference to the one that matches it exactly. + """ + lookup: dict[str, str] = {} + for id_ in ids: + lookup.setdefault(nfc(id_), id_) + for id_ in ids: + lookup[id_] = id_ + return lookup + + def resolve(name: str, ids: dict[str, str]) -> str | None: + """The id ``name`` names, exactly or after normalizing; None if no id.""" + exact = ids.get(name) + return exact if exact is not None else ids.get(nfc(name)) + + # Range integrity over the merged view (inline + companions). A finding + # about a range is addressed to the companion that defines it, which is not + # necessarily the document being validated. all_ranges = lexicon.all_ranges() + ranges_paths = { + range_.id: ranges_file.path + for ranges_file in lexicon.ranges_files.values() + for range_ in ranges_file.ranges + if all_ranges.get(range_.id) is range_ # an inline range wins the merge + } + # Each range's element lookup is kept for the value checks below, which + # would otherwise rebuild it per trait. Only ranges that enumerate elements + # can confirm a parent or a value. + lookups: dict[str, dict[str, str]] = {} for range_ in all_ranges.values(): - element_ids = {element.id for element in range_.elements} + if not range_.elements: + continue + lookups[range_.id] = element_ids = id_lookup([e.id for e in range_.elements]) for element in range_.elements: - if element.parent and element.parent not in element_ids: + if not element.parent: + continue + target = resolve(element.parent, element_ids) + if target is None: yield Problem( "error", "range-parent", f"range {range_.id!r}: range-element {element.id!r} has " f"parent {element.parent!r} which is not a sibling id", - file=file, + file=ranges_paths.get(range_.id) or file, ) + elif target != element.parent: + mismatched.setdefault((range_.id, "range-element id", target), element.parent) + + # The name keying a range -- a header id, a trait name -- is written + # separately from the range's own id, so it resolves the same way. + range_ids = id_lookup(lookups.keys()) + + def range_named(name: str) -> str | None: + """The id of the range ``name`` keys, or None if no such range + enumerates elements. Records a name that reached one by normalizing. + """ + range_id = resolve(name, range_ids) + if range_id is not None and range_id != name: + mismatched.setdefault((range_id, "range id", range_id), name) + return range_id + + # Whether a header range id and the companion's own id agree is a fact about + # the document, not about there being a file to look for -- so it is settled + # here, not inside the file check below, which a lexicon with no path skips. + header_ranges = {range_.id: range_named(range_.id) for range_ in lexicon.header.ranges} # Header references (relative) that resolve to no companion. # Absolute/file:// hrefs are ones FLEx writes knowing they will not resolve @@ -444,8 +520,7 @@ def _main_doc_guids() -> Iterator[tuple[str, str, str | None, int | None]]: relative = _normalize_href(range_.href) if relative is None: continue - resolved = all_ranges.get(range_.id) - if resolved is not None and resolved.elements: + if header_ranges[range_.id] is not None: continue # supplied by a sibling companion instead if not (base / relative).is_file(): yield Problem( @@ -456,37 +531,25 @@ def _main_doc_guids() -> Iterator[tuple[str, str, str | None, int | None]]: file=lexicon.path, ) - # Undefined range values: every grammatical-info (sense, reversal, and - # reversal main chains) against the grammatical-info range; every trait - # anywhere in the entry whose name matches a known range — not just entry- - # and sense-direct ones, since real FLEx exports nest traits like - # is-primary/complex-form-type inside and morph-type inside - # (_iter_traits/_iter_grammatical_infos walk the whole entry). - # Only ranges that actually enumerate elements can confirm a value; empty - # values skipped (FLEx writes them). Comparison is NFC-normalized: FLEx - # writes the .lift in NFC but the companion .lift-ranges in NFD within the - # same export. - def nfc(value: str) -> str: - return unicodedata.normalize("NFC", value) - - def defined(range_id: str) -> set[str] | None: - range_: Range | None = all_ranges.get(range_id) - if range_ is None or not range_.elements: - return None - return {nfc(element.id) for element in range_.elements} - - grammatical_values = defined("grammatical-info") + # Undefined range values: every grammatical-info and every trait whose + # name matches a known range, anywhere in the entry. Empty values are + # skipped -- FLEx writes them. + grammatical_range = range_named("grammatical-info") for index, entry in enumerate(lexicon.entries): - checks: list[tuple[str, str, set[str]]] = [] # (label, value, allowed) - for info in _iter_grammatical_infos(entry): - if info.value and grammatical_values is not None: - checks.append(("grammatical-info", info.value, grammatical_values)) + checks: list[tuple[str, str, str]] = [] # (label, range id, value) + if grammatical_range is not None: + for info in _iter_grammatical_infos(entry): + if info.value: + checks.append(("grammatical-info", grammatical_range, info.value)) for trait in _iter_traits(entry): - allowed = defined(trait.name) - if allowed is not None and trait.value: - checks.append((f"trait {trait.name!r}", trait.value, allowed)) - for label, value, allowed in checks: - if nfc(value) not in allowed: + # Resolved before the value guard: a trait name that reaches its range only + # by normalizing is worth reporting even without a value. + range_id = range_named(trait.name) + if range_id is not None and trait.value: + checks.append((f"trait {trait.name!r}", range_id, trait.value)) + for label, range_id, value in checks: + target = resolve(value, lookups[range_id]) + if target is None: yield Problem( "warning", "undefined-range-value", @@ -496,6 +559,19 @@ def defined(range_id: str) -> set[str] | None: guid=entry.guid, line=at(index), ) + elif target != value: + mismatched.setdefault((range_id, "range-element id", target), value) + + # The two spellings render identically, so name them by code point rather + # than with the !r other messages use. + for (range_id, kind, id_), reference in sorted(mismatched.items()): + yield Problem( + "warning", + "normalization-mismatch", + f"range {range_id!r}: {kind} {id_!a} is referenced as {reference!a}; " + "they match only under Unicode NFC normalization", + file=ranges_paths.get(range_id) or file, + ) # Missing media files. for media_ref in lexicon.missing_media(): diff --git a/tests/corpus/PROVENANCE.md b/tests/corpus/PROVENANCE.md index 2a3df46..699026f 100644 --- a/tests/corpus/PROVENANCE.md +++ b/tests/corpus/PROVENANCE.md @@ -156,6 +156,31 @@ with one `HashSet` and would flag this too — so sil-lift's `duplicate-guid` check reports it as an error, matching that behavior. `tests/test_validate.py` locks in the exact counts found (5 in AllFLExFields, 37 in Sango). +## Known Unicode normalization asymmetry in real fixtures + +FLEx holds strings as NFD and normalizes them to NFC on export, but a few writes +bypass its normalizing helper and emit the NFD unchanged. In +`large/sango/sango.lift-ranges` the `grammatical-info`, `from-part-of-speech`, +and `lexical-relation` range-element `id`s are NFD while everything around them +— their `label`/`abbrev`/`description`, the `parent` attribute of those same +elements, the `semantic-domain-ddp4`/`translation-type`/`usage-type` ids, and +every value in `large/sango/sango.lift` — is NFC. + +So one name appears in two spellings within a single element: the part of +speech whose id decomposes as `Comple` \+ U+0301 \+ `ments` is named by a +child element's `parent` attribute with the precomposed U+00E9 instead. Both +`range-parent` and `undefined-range-value` therefore compare under NFC +normalization; without it, this fixture reports two dangling parents and an +extra undefined part of speech that are artifacts of the encoding, not defects. +The one genuine `undefined-range-value` left in Sango is the part of speech +`prenom`, which no range defines in any normalization. + +What did match only after normalizing is still reported, as a +`normalization-mismatch` warning per range-element id: 2 parent links and 80 +grammatical-info values reach 5 ids in Sango, one of them under both aliases +of the part-of-speech list, so 6 warnings — and none in the other fixtures. +`negative/nfd-range-ids.lift` is the hand-authored version of the same shape. + ## generated/ — synthetic large files (not committed) Produced by `tests/tools/generate_large.py` for streaming/perf tests; @@ -167,8 +192,11 @@ Each file carries an XML comment documenting its defect and the expected Problem code: `duplicate-guid`, `dangling-ref`, `range-parent`, `undefined-range-value` (2 warnings \+ a clean control entry), `duplicate-form-lang` (the Schematron-only rule), `schema-invalid` -(structural), `missing-media/` (a folder fixture), and `flex-quirks` -(URI quirks that must yield warnings, never schema errors). +(structural), `missing-media/` (a folder fixture), `flex-quirks` +(URI quirks that must yield warnings, never schema errors), and +`nfd-range-ids` (a `.lift` \+ `.lift-ranges` pair carrying FLEx's +normalization asymmetry: NFD ids, NFC references, and one parent that +dangles in every normalization). `schema-invalid.lift` and `flex-quirks.lift` are raw-RNG-invalid (the latter only under libxml2's anyURI check) and appear in the corpus test's expected-invalid list. diff --git a/tests/corpus/negative/nfd-range-ids.lift b/tests/corpus/negative/nfd-range-ids.lift new file mode 100644 index 0000000..a933045 --- /dev/null +++ b/tests/corpus/negative/nfd-range-ids.lift @@ -0,0 +1,20 @@ + + + +
+ + + +
+ +
de
 + + + diff --git a/tests/corpus/negative/nfd-range-ids.lift-ranges b/tests/corpus/negative/nfd-range-ids.lift-ranges new file mode 100644 index 0000000..ac0a6f1 --- /dev/null +++ b/tests/corpus/negative/nfd-range-ids.lift-ranges @@ -0,0 +1,14 @@ + + + + + + + + + + + + diff --git a/tests/test_validate.py b/tests/test_validate.py index 6249154..6c5616e 100644 --- a/tests/test_validate.py +++ b/tests/test_validate.py @@ -1,3 +1,4 @@ +import unicodedata from pathlib import Path import pytest @@ -17,6 +18,11 @@ def codes(problems: list[Problem]) -> set[tuple[str, str]]: return {(p.level, p.code) for p in problems} +def nfd(value: str) -> str: + """The decomposed spelling, as FLEx leaves the ids that skip its normalizer.""" + return unicodedata.normalize("NFD", value) + + def test_duplicate_guid_is_error_with_addressing() -> None: problems = problems_for(NEGATIVE_DIR / "duplicate-guid.lift") (problem,) = problems @@ -41,6 +47,138 @@ def test_range_parent_integrity() -> None: assert "Nooun" in problem.message +def test_range_parent_tolerates_flex_normalization_asymmetry() -> None: + # Regression: FLEx writes a grammatical-info range-element id straight from + # its NFD in-memory string but normalizes the parent attribute to NFC (see + # PROVENANCE.md), so the two spellings of one name differ within a single + # element. The parent link is sound; only the encoding differs. + parent_name = "Compl\u00e9ments" # NFC, the form FLEx writes a parent in + lexicon = sil_lift.Lexicon() + ranges = sil_lift.RangesFile() + range_ = ranges.add_range("grammatical-info") + range_.add_element(nfd(parent_name)) # ids skip FLEx's normalizer + range_.add_element(nfd("Compl\u00e9ment du lieu"), parent=parent_name) + lexicon.add_ranges_file(ranges, href="x.lift-ranges") + entry = sil_lift.Entry(id="e1", guid="bbbbbbbb-1111-4444-8888-bbbbbbbbbbbb") + entry.lexical_unit["en"] = "e1" + lexicon.entries.append(entry) + assert [p for p in lexicon.iter_problems() if p.code == "range-parent"] == [] + + +def test_normalization_mismatch_prefers_an_exactly_matching_sibling() -> None: + # A range may hold both spellings of one name -- ids are unique as strings, + # and FLEx normalizes some writes and not others. Every reference here + # matches a sibling exactly, so none of them needed normalizing. + name = "Preposi\u00e7\u00e3o" + lexicon = sil_lift.Lexicon() + ranges = sil_lift.RangesFile() + range_ = ranges.add_range("grammatical-info") + range_.add_element(nfd(name)) + range_.add_element(name) + range_.add_element("Associativo", parent=nfd(name)) + range_.add_element("Prepositional phrase", parent=name) + lexicon.add_ranges_file(ranges, href="x.lift-ranges") + entry = sil_lift.Entry(id="e1", guid="cccccccc-1111-4444-8888-cccccccccccc") + entry.lexical_unit["en"] = "e1" + entry.senses.append( + sil_lift.Sense(id="s1", grammatical_info=sil_lift.GrammaticalInfo(nfd(name))) + ) + lexicon.entries.append(entry) + assert list(lexicon.iter_problems()) == [] + + +def test_trait_name_reaches_a_range_id_in_another_normalization() -> None: + # A custom FLEx list becomes a range whose id is the list name and traits + # whose name is that same string -- separate writes, so they can differ in + # normalization exactly as an id and the values naming it do. The range has + # to be resolved for its values to be checked at all: an unresolved name + # looks like a trait no range keys, which is silently accepted. + name = "Catégorie" + lexicon = sil_lift.Lexicon() + ranges = sil_lift.RangesFile() + range_ = ranges.add_range(nfd(name)) + range_.add_element("Nom") + lexicon.add_ranges_file(ranges, href="x.lift-ranges") + entry = sil_lift.Entry(id="e1", guid="dddddddd-1111-4444-8888-dddddddddddd") + entry.lexical_unit["en"] = "e1" + entry.traits.append(sil_lift.Trait(name=name, value="Verbe")) + lexicon.entries.append(entry) + problems = list(lexicon.iter_problems()) + assert codes(problems) == { + ("warning", "undefined-range-value"), + ("warning", "normalization-mismatch"), + } + (mismatch,) = [p for p in problems if p.code == "normalization-mismatch"] + assert "range id" in mismatch.message + assert "Cate\\u0301gorie" in mismatch.message + assert "Cat\\xe9gorie" in mismatch.message + + +def test_header_range_id_reaches_a_companion_id_in_another_normalization() -> None: + # Regression: the resolution used to sit inside the dangling-href check, + # which runs only for a lexicon read from disk, so this went unreported. + name = "Catégorie" + lexicon = sil_lift.Lexicon() + ranges = sil_lift.RangesFile() + ranges.add_range(nfd(name)).add_element("Nom") + lexicon.add_ranges_file(ranges, href="x.lift-ranges") # header id as written + lexicon.header.ranges.append(sil_lift.Range(id=name, href="x.lift-ranges")) + entry = sil_lift.Entry(id="e1", guid="eeeeeeee-1111-4444-8888-eeeeeeeeeeee") + entry.lexical_unit["en"] = "e1" + lexicon.entries.append(entry) + problems = list(lexicon.iter_problems()) + assert codes(problems) == {("warning", "normalization-mismatch")} + assert "range id" in problems[0].message + + +def test_nfd_ids_warn_once_and_still_flag_the_real_dangling_parent(tmp_path: Path) -> None: + path = NEGATIVE_DIR / "nfd-range-ids.lift" + problems = problems_for(path) + assert codes(problems) == {("error", "range-parent"), ("warning", "normalization-mismatch")} + (dangling,) = [p for p in problems if p.code == "range-parent"] + assert "Preposicao" in dangling.message + # The element is in the companion, so that is where the error is addressed. + assert dangling.file is not None and dangling.file.suffix == ".lift-ranges" + # Two parent attributes and one grammatical-info value resolve to the same + # id: one finding, against the companion the id lives in, naming both + # spellings by code point (they are indistinguishable rendered). + (mismatch,) = [p for p in problems if p.code == "normalization-mismatch"] + assert mismatch.file is not None and mismatch.file.suffix == ".lift-ranges" + assert "Preposic\\u0327a\\u0303o" in mismatch.message + assert "Preposi\\xe7\\xe3o" in mismatch.message + # Normalization belongs to the comparison only: the mixed forms survive. + sil_lift.Lexicon.load(path).save(tmp_path / path.name) + for name in (path.name, "nfd-range-ids.lift-ranges"): + assert (tmp_path / name).read_bytes() == (NEGATIVE_DIR / name).read_bytes(), name + + +def test_one_id_referenced_in_two_spellings_still_warns_once() -> None: + # The dedup is per id, not per spelling: references that differ from the id + # and from each other still collapse into one warning. Which of them the + # message names is left unasserted -- the contract is the count, not the + # order the references happen to be reached in. + name = "ṩ" # s with dot below and dot above, precomposed + # A third spelling: the same two marks in the other order, matching neither + # the id nor the parent link below (nfd() orders them canonically). + other_order = "ṩ" + lexicon = sil_lift.Lexicon() + ranges = sil_lift.RangesFile() + range_ = ranges.add_range("grammatical-info") + range_.add_element(name) + range_.add_element("Enclitic", parent=nfd(name)) + lexicon.add_ranges_file(ranges, href="x.lift-ranges") + entry = sil_lift.Entry(id="e1", guid="ffffffff-1111-4444-8888-ffffffffffff") + entry.lexical_unit["en"] = "e1" + entry.senses.append( + sil_lift.Sense(id="s1", grammatical_info=sil_lift.GrammaticalInfo(other_order)) + ) + lexicon.entries.append(entry) + problems = list(lexicon.iter_problems()) + assert codes(problems) == {("warning", "normalization-mismatch")} + assert len(problems) == 1 + assert "range-element id '\\u1e69'" in problems[0].message + + def test_undefined_range_values_are_warnings() -> None: problems = problems_for(NEGATIVE_DIR / "undefined-range-value.lift") assert codes(problems) == {("warning", "undefined-range-value")} @@ -199,10 +337,15 @@ def test_sango_real_defects_are_found() -> None: by_code: dict[str, int] = {} for problem in problems: by_code[problem.code] = by_code.get(problem.code, 0) + 1 - # Two genuinely dangling range-element parents + one undefined POS value - # ('prenom') in the real export; NFC-normalization keeps the count at 1. - assert by_code.get("range-parent") == 2 + # One undefined POS value ('prenom') in the real export; NFC-normalization + # keeps the count at 1. No range-parent finding: the two parent links that + # differ from their target's spelling are FLEx's NFD ids under an NFC + # parent (see PROVENANCE.md), not dangling references. Those two and 80 + # NFC grammatical-info values reach 5 NFD ids; the aliased POS list below + # holds one of them under each of its two range ids, so 6 warnings. + assert by_code.get("range-parent") is None assert by_code.get("undefined-range-value") == 1 + assert by_code.get("normalization-mismatch") == 6 assert by_code.get("schema", 0) > 0 # companion's trait/field extensions # 37 real duplicate guids: FLEx aliases its POS list under both # "grammatical-info" and "from-part-of-speech" (same range-element guids