Compare range-element parents under NFC

CHANGELOG.md

@@ -73,9 +73,14 @@ releases may contain breaking changes.
   ranges/range-elements within their own document — matching the C#
   `Validator`'s document-wide guid scan), dangling-ref, range-parent,
   undefined-range-value (every grammatical-info and range-keyed trait
-  reachable from an entry, however nested \+ NFC-normalized),
+  reachable from an entry, however nested), normalization-mismatch,
   duplicate-form-lang, missing-media, dangling-ranges-href, and (opt-in via
-  `require_ids`) missing-id.
+  `require_ids`) missing-id. Names are resolved against range and
+  range-element ids under NFC normalization: FLEx can write an id in NFD while
+  the trait name, value, or `parent` referring to it is NFC. What matched only
+  after normalizing is reported as a normalization-mismatch warning, once per
+  id, so the encoding split stays visible to consumers that compare raw
+  strings.
 - Canonical sort: `Lexicon.sort()` / `RangesFile.sort()` (entries by
   case-folded guid/id, ranges/range-elements by id, field definitions by tag;
   informed by the C# LiftSorter, locale-independent) and

docs/en/csharp-differences.md

@@ -31,7 +31,7 @@ The C# `Validator` runs one RELAX NG pass and reports the first errors as string
 
 - **Invalid URIs are warnings, not errors.** The C# RELAX NG engine never enforced the `anyURI` datatype, so FieldWorks (FLEx) has been writing `file://C:/...` hrefs into real lexicons for years. Rejecting those files would flag virtually every FLEx export.
 - **Schematron rules are enforced** (as semantic checks): duplicate form languages and similar co-constraints in the LIFT grammar were silently ignored by both C# and raw lxml validation.
-- **Cross-file comparisons are Unicode-normalized**, because FLEx writes the `.lift` in NFC and the companion `.lift-ranges` in NFD.
+- **Range id comparisons are Unicode-normalized** (NFC), because FLEx's own export is not internally consistent: it normalizes to NFC on the way out, but a few writes bypass that step and emit the NFD it holds in memory. A `grammatical-info` or `lexical-relation` range-element `id` can therefore be NFD while its labels, the `parent` attribute on that same element, and the `.lift` value referring to it are all NFC. References that resolve only after normalizing are reported as `normalization-mismatch` warnings — a check with no C# counterpart — so the encoding split stays visible to anyone whose own comparisons are exact.
 
 sil-lift also validates the `.lift-ranges` companions of a loaded lexicon against a schema for standalone ranges documents (vendored from `lift-standard` alongside the base LIFT grammar) — every tracked external ranges file is checked whenever the `.lift` is validated — with no such schema (or check) in the C# world. (There is no entry point for validating a `.lift-ranges` file on its own, detached from a `.lift`.)
 

docs/en/guides/lift-export-interop.md

@@ -92,7 +92,7 @@ The companion carries each range's full definition. Values are `<range-element>`
 </lift-ranges>
 ```
 
-An entry then refers to a value by id: a sense's part of speech is `<grammatical-info value="Noun"/>`, and a semantic domain is `<trait name="semantic-domain-ddp4" value="1.6.1.2"/>`. `sil-lift validate` warns (`undefined-range-value`) when a value isn't defined in its range and errors (`range-parent`) when a `parent` isn't a sibling id — so emit the ranges your data actually uses. See also [Ranges and media](folder-media.md).
+An entry then refers to a value by id: a sense's part of speech is `<grammatical-info value="Noun"/>`, and a semantic domain is `<trait name="semantic-domain-ddp4" value="1.6.1.2"/>`. `sil-lift validate` warns (`undefined-range-value`) when a value isn't defined in its range and errors (`range-parent`) when a `parent` isn't a sibling id — so emit the ranges your data actually uses. Those comparisons are NFC-normalized, so an id and the value or `parent` referring to it may differ in Unicode normalization — that difference is a `normalization-mismatch` warning rather than an error, but write one consistent normalization if you can: consumers that compare raw strings will not resolve those references. See also [Ranges and media](folder-media.md).
 
 If you build the export in Python, `Lexicon.add_ranges_file()`, `RangesFile.add_range()`, and `Range.add_element()` construct the companion and add the header references for you; `open_writer(..., ranges=...)` does the same on the streaming path.
 

docs/en/guides/validate.md

@@ -18,19 +18,22 @@ lex = sil_lift.load("dictionary.lift")
 problems = list(lex.iter_problems())
 ```
 
-Each `Problem` carries `level` (`"error"`/`"warning"`), a stable `code`, `message`, and an address: `file`, `entry_id`, `guid`, `line`.
+Each `Problem` carries `level` (`"error"`/`"warning"`), a stable `code`, `message`, and an address: `file`, plus `entry_id`, `guid`, and `line` when the finding belongs to one entry. Findings about a document rather than an entry — `range-parent`, `normalization-mismatch`, `dangling-ranges-href` — carry the file alone (for a finding about a range, the `.lift-ranges` companion that defines it), and the unset fields are `None` (`null` in `--format json`, where every key is always present).
 
 ## The layers
 
 1. **RELAX NG** against the LIFT 0.13 grammar (vendored from lift-standard — a byte-identical copy committed into this package).
 2. **Ranges schema** — this project's `lift-ranges-0.13.rng` — over every tracked `.lift-ranges` companion.
-3. **Semantic checks** the grammar cannot express: `duplicate-guid`, `dangling-ref`, `range-parent`, `undefined-range-value`, `duplicate-form-lang`, `missing-media`.
+3. **Semantic checks** the grammar cannot express: `duplicate-guid`, `dangling-ref`, `range-parent`, `undefined-range-value`, `normalization-mismatch`, `duplicate-form-lang`, `missing-media`.
 
 ## Real-world FieldWorks (FLEx) output
 
 FieldWorks systematically writes some content that strict tooling rejects. Here is sil-lift's policy, so that real lexicons validate usefully:
 
 - `file://C:/...` hrefs (invalid URIs) are reported as **warnings** (`uri-not-rfc`), not schema errors — the C# validator never rejected them.
 - Legally interleaved children (e.g. `field, note, field, note` in a sense) are **not** flagged, working around a false positive in libxml2.
-- Range values are compared under Unicode NFC normalization — FLEx writes the `.lift` in NFC but the `.lift-ranges` in NFD within the same export.
 - FLEx's `trait`/`field` extensions inside `range-element` **are** reported (schema errors against the ranges schema): they are genuine spec deviations.
+- Names are resolved against range and range-element `id`s under Unicode **NFC normalization** — `parent` links, range values, and the `trait` name or header `range` id that keys a range. FLEx normalizes to NFC on export, but a few writes bypass that step, so a range-element `id` can be NFD while its labels, its own `parent`, and the `.lift` values naming it are NFC.
+    - Compared exactly, a sound export looks broken — and a range whose `id` is spelled the other way goes unchecked entirely, since a trait name that reaches no range is silently accepted.
+    - A name that matched only after normalizing is reported as a `normalization-mismatch` **warning**, once per id however many references differ, addressed to the file that defines it. The data is sound, but a consumer comparing raw strings — a Send/Receive merge, say — will not resolve those references.
+    - The ids are never rewritten: the file keeps the spellings it came with.

src/sil_lift/_validate.py

@@ -20,14 +20,24 @@
      left untouched.
 2. The vendored ``lift-ranges-0.13.rng`` over each tracked ``.lift-ranges``
    companion.
-3. Semantic checks the grammar cannot express: duplicate GUIDs (entries, and
-   ranges/range-elements within their own document), dangling ``relation/@ref``
-   and ``variant/@ref``, ``range-element/@parent`` integrity, undefined range
-   values (every grammatical-info and range-keyed trait anywhere in the entry,
-   however deeply nested), duplicate form languages (the RNG's Schematron
-   rule, which lxml ignores), missing media files, header ``range/@href``
-   references that resolve to no companion, and — opt-in — entries/senses
-   missing a stable id.
+3. Semantic checks that the grammar cannot express, one ``Problem`` code each:
+
+   - ``duplicate-guid`` — a guid reused, among entries or among the
+     ranges/range-elements of one document.
+   - ``dangling-ref`` — a ``relation/@ref`` or ``variant/@ref`` matching no
+     entry or sense.
+   - ``range-parent`` — a ``range-element/@parent`` no sibling id defines.
+   - ``undefined-range-value`` — a grammatical-info or range-keyed trait value
+     the range does not list, checked however deeply nested in the entry.
+   - ``normalization-mismatch`` — a name that reaches the range or
+     range-element id it refers to only under Unicode NFC normalization,
+     because FLEx writes some ids in NFD and the references to them in NFC.
+   - ``duplicate-form-lang`` — two forms in one multitext sharing a language
+     (the RNG's Schematron rule, which lxml ignores).
+   - ``missing-media`` — a referenced audio or picture file not on disk.
+   - ``dangling-ranges-href`` — a header ``range/@href`` resolving to no
+     companion file.
+   - ``missing-id`` — opt-in: an entry without a guid, a sense without an id.
 """
 
 from __future__ import annotations
@@ -45,7 +55,7 @@
 
 if TYPE_CHECKING:
     import os
-    from collections.abc import Iterator
+    from collections.abc import Collection, Iterator
 
     from ._header import Range
     from ._model import Entry, Sense
@@ -62,8 +72,8 @@ class Problem:
 
     level: Literal["error", "warning"]
     code: str  # "schema", "duplicate-guid", "dangling-ref", "range-parent",
-    # "undefined-range-value", "duplicate-form-lang", "missing-media",
-    # "uri-not-rfc", "dangling-ranges-href", "missing-id"
+    # "undefined-range-value", "normalization-mismatch", "duplicate-form-lang",
+    # "missing-media", "uri-not-rfc", "dangling-ranges-href", "missing-id"
     message: str
     file: Path | None = None
     entry_id: str | None = None
@@ -417,19 +427,85 @@ def _main_doc_guids() -> Iterator[tuple[str, str, str | None, int | None]]:
                     line=at(index),
                 )
 
-    # Range integrity over the merged view (inline + companions).
+    # FLEx writes some ids in NFD but every reference to them in NFC, so
+    # resolving a name to an id has to compare both forms.
+    def nfc(value: str) -> str:
+        return unicodedata.normalize("NFC", value)
+
+    # Names that resolved only after normalizing, keyed by what they resolved
+    # to: (range id, what the id is, id as written) -> one differing reference.
+    # Warning per id rather than per reference keeps a real export to 6
+    # findings instead of 82.
+    mismatched: dict[tuple[str, str, str], str] = {}
+
+    def id_lookup(ids: Collection[str]) -> dict[str, str]:
+        """What each spelling of these ids resolves to: the ids as written,
+        keyed by both themselves and their NFC form. An id present as written
+        maps to itself, so a collection holding two normalizations of one name
+        resolves an exact reference to the one that matches it exactly.
+        """
+        lookup: dict[str, str] = {}
+        for id_ in ids:
+            lookup.setdefault(nfc(id_), id_)
+        for id_ in ids:
+            lookup[id_] = id_
+        return lookup
+
+    def resolve(name: str, ids: dict[str, str]) -> str | None:
+        """The id ``name`` names, exactly or after normalizing; None if no id."""
+        exact = ids.get(name)
+        return exact if exact is not None else ids.get(nfc(name))
+
+    # Range integrity over the merged view (inline + companions). A finding
+    # about a range is addressed to the companion that defines it, which is not
+    # necessarily the document being validated.
     all_ranges = lexicon.all_ranges()
+    ranges_paths = {
+        range_.id: ranges_file.path
+        for ranges_file in lexicon.ranges_files.values()
+        for range_ in ranges_file.ranges
+        if all_ranges.get(range_.id) is range_  # an inline range wins the merge
+    }
+    # Each range's element lookup is kept for the value checks below, which
+    # would otherwise rebuild it per trait. Only ranges that enumerate elements
+    # can confirm a parent or a value.
+    lookups: dict[str, dict[str, str]] = {}
     for range_ in all_ranges.values():
-        element_ids = {element.id for element in range_.elements}
+        if not range_.elements:
+            continue
+        lookups[range_.id] = element_ids = id_lookup([e.id for e in range_.elements])
         for element in range_.elements:
-            if element.parent and element.parent not in element_ids:
+            if not element.parent:
+                continue
+            target = resolve(element.parent, element_ids)
+            if target is None:
                 yield Problem(
                     "error",
                     "range-parent",
                     f"range {range_.id!r}: range-element {element.id!r} has "
                     f"parent {element.parent!r} which is not a sibling id",
-                    file=file,
+                    file=ranges_paths.get(range_.id) or file,
                 )
+            elif target != element.parent:
+                mismatched.setdefault((range_.id, "range-element id", target), element.parent)
+
+    # The name keying a range -- a header id, a trait name -- is written
+    # separately from the range's own id, so it resolves the same way.
+    range_ids = id_lookup(lookups.keys())
+
+    def range_named(name: str) -> str | None:
+        """The id of the range ``name`` keys, or None if no such range
+        enumerates elements. Records a name that reached one by normalizing.
+        """
+        range_id = resolve(name, range_ids)
+        if range_id is not None and range_id != name:
+            mismatched.setdefault((range_id, "range id", range_id), name)
+        return range_id
+
+    # Whether a header range id and the companion's own id agree is a fact about
+    # the document, not about there being a file to look for -- so it is settled
+    # here, not inside the file check below, which a lexicon with no path skips.
+    header_ranges = {range_.id: range_named(range_.id) for range_ in lexicon.header.ranges}
 
     # Header <range href> references (relative) that resolve to no companion.
     # Absolute/file:// hrefs are ones FLEx writes knowing they will not resolve
@@ -444,8 +520,7 @@ def _main_doc_guids() -> Iterator[tuple[str, str, str | None, int | None]]:
             relative = _normalize_href(range_.href)
             if relative is None:
                 continue
-            resolved = all_ranges.get(range_.id)
-            if resolved is not None and resolved.elements:
+            if header_ranges[range_.id] is not None:
                 continue  # supplied by a sibling companion instead
             if not (base / relative).is_file():
                 yield Problem(
@@ -456,37 +531,25 @@ def _main_doc_guids() -> Iterator[tuple[str, str, str | None, int | None]]:
                     file=lexicon.path,
                 )
 
-    # Undefined range values: every grammatical-info (sense, reversal, and
-    # reversal main chains) against the grammatical-info range; every trait
-    # anywhere in the entry whose name matches a known range — not just entry-
-    # and sense-direct ones, since real FLEx exports nest traits like
-    # is-primary/complex-form-type inside <relation> and morph-type inside
-    # <variant> (_iter_traits/_iter_grammatical_infos walk the whole entry).
-    # Only ranges that actually enumerate elements can confirm a value; empty
-    # values skipped (FLEx writes them). Comparison is NFC-normalized: FLEx
-    # writes the .lift in NFC but the companion .lift-ranges in NFD within the
-    # same export.
-    def nfc(value: str) -> str:
-        return unicodedata.normalize("NFC", value)
-
-    def defined(range_id: str) -> set[str] | None:
-        range_: Range | None = all_ranges.get(range_id)
-        if range_ is None or not range_.elements:
-            return None
-        return {nfc(element.id) for element in range_.elements}
-
-    grammatical_values = defined("grammatical-info")
+    # Undefined range values: every grammatical-info and every trait whose
+    # name matches a known range, anywhere in the entry. Empty values are
+    # skipped -- FLEx writes them.
+    grammatical_range = range_named("grammatical-info")
     for index, entry in enumerate(lexicon.entries):
-        checks: list[tuple[str, str, set[str]]] = []  # (label, value, allowed)
-        for info in _iter_grammatical_infos(entry):
-            if info.value and grammatical_values is not None:
-                checks.append(("grammatical-info", info.value, grammatical_values))
+        checks: list[tuple[str, str, str]] = []  # (label, range id, value)
+        if grammatical_range is not None:
+            for info in _iter_grammatical_infos(entry):
+                if info.value:
+                    checks.append(("grammatical-info", grammatical_range, info.value))
         for trait in _iter_traits(entry):
-            allowed = defined(trait.name)
-            if allowed is not None and trait.value:
-                checks.append((f"trait {trait.name!r}", trait.value, allowed))
-        for label, value, allowed in checks:
-            if nfc(value) not in allowed:
+            # Resolved before the value guard: a trait name that reaches its range only
+            # by normalizing is worth reporting even without a value.
+            range_id = range_named(trait.name)
+            if range_id is not None and trait.value:
+                checks.append((f"trait {trait.name!r}", range_id, trait.value))
+        for label, range_id, value in checks:
+            target = resolve(value, lookups[range_id])
+            if target is None:
                 yield Problem(
                     "warning",
                     "undefined-range-value",
@@ -496,6 +559,19 @@ def defined(range_id: str) -> set[str] | None:
                     guid=entry.guid,
                     line=at(index),
                 )
+            elif target != value:
+                mismatched.setdefault((range_id, "range-element id", target), value)
+
+    # The two spellings render identically, so name them by code point rather
+    # than with the !r other messages use.
+    for (range_id, kind, id_), reference in sorted(mismatched.items()):
+        yield Problem(
+            "warning",
+            "normalization-mismatch",
+            f"range {range_id!r}: {kind} {id_!a} is referenced as {reference!a}; "
+            "they match only under Unicode NFC normalization",
+            file=ranges_paths.get(range_id) or file,
+        )
 
     # Missing media files.
     for media_ref in lexicon.missing_media():