diff --git a/AGENTS.md b/AGENTS.md index 9b7f0115..c061a25a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -129,12 +129,12 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p ### Configuration layer (`nameparser/config/`) -Most modules define a plain Python set of known name pieces; `capitalization.py` and `regexes.py` define dicts: +Most modules define a `frozenset` of known name pieces; `capitalization.py` and `regexes.py` define dicts. The SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` on any of them, so a default word list is changed by configuring an object — a private `Constants` for `HumanName`, a `Lexicon` for the 2.0 API — never by editing the constant. A union of a `frozenset` with a set literal is still a `frozenset` (`TITLES`, `PARTICLES`), so the derived sets are frozen too. `CONSTANTS`/`Constants` still hand out mutable `SetManager`s; the freeze is on the module set constants they copy from. **Neither dict was frozen, and `CAPITALIZATION_EXCEPTIONS` is not covered by anything else either.** `REGEXES` is a compiled-pattern table rather than vocabulary and was never in #293's scope; `CAPITALIZATION_EXCEPTIONS` is vocabulary-shaped and is a decided, in-scope exemption. So the split-default hazard the freeze closes is still live for it, measured on 2.2: `CAPITALIZATION_EXCEPTIONS['phd'] = 'PhD'` reaches a freshly built `Constants` and neither the cached `Lexicon.default()` nor the shared `CONSTANTS`. Same advice — configure the object (`constants.capitalization_exceptions[...]`, or `dataclasses.replace(lexicon, capitalization_exceptions=...)`). `tests/v2/test_contracts.py::test_every_vocabulary_constant_is_frozen` names both dicts as explicit exemptions rather than letting its `isinstance` filter drop them. -- `titles.py` — `TITLES` (prenominals) and `FIRST_NAME_TITLES` (e.g. "Sir", which treat the following name as first, not last) -- `suffixes.py` — `SUFFIX_ACRONYMS` (with periods, e.g. "M.D.") and `SUFFIX_NOT_ACRONYMS` (e.g. "Jr."), plus `GLUED_HONORIFICS` (#308), the subset of `SUFFIX_NOT_ACRONYMS` the peel may split off the END of a name token — a separate, harsher set, since the glued position has no writer-drawn boundary to lean on -- `prefixes.py` — `PREFIXES` (lastname particles, e.g. "de", "van") -- `bound_first_names.py` — `BOUND_FIRST_NAMES` (bound given-name prefixes, e.g. "abdul", "abu"); `_join_bound_first_name` joins the first non-title piece to its following piece before the main assignment loop +- `titles.py` — `TITLES` (prenominals) and `GIVEN_NAME_TITLES` (e.g. "Sir", which treat the following name as given, not family) +- `suffixes.py` — `SUFFIX_ACRONYMS` (with periods, e.g. "M.D.") and `SUFFIX_WORDS` (e.g. "Jr."), plus `GLUED_HONORIFICS` (#308), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher set, since the glued position has no writer-drawn boundary to lean on +- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (under the DEFAULT given-first order a name starting with one is all surname: "de Mesnil" — but that is `name_order`'s half of the sentence, not this set's, and `Policy(name_order=FAMILY_FIRST)` reads the same input as family "de", given "Mesnil"; what the set decides under either order is that a leading particle outside it records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either +- `bound_given_names.py` — `BOUND_GIVEN_NAMES` (bound given-name prefixes, e.g. "abdul", "abu"); a group-stage rule joins the first non-title piece to its following piece before roles are assigned (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles - `maiden_markers.py` — `MAIDEN_MARKERS` (e.g. "née", "geb.") routing the following name to `maiden` - `surnames.py` — `KOREAN_SURNAMES`, the census list the 2.0 API splits unspaced hangul on (#271). With `maiden_markers.py` it is one of the two data modules `Constants` has **no** attribute for: both reach the parse only through `Constants._snapshot()` → `Lexicon`, so the v1 surface stays frozen and there is no v1 knob to turn either off (the opt-out is the 2.0 `Policy`) @@ -205,11 +205,11 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Unknown-key attribute access on `TupleManager`/`RegexTupleManager` warns (1.4, #256) — 2.0 note: the warning became a hard `AttributeError` naming the known keys (shim `TupleManager`); the paragraph below describes the deleted v1 machinery and applies only when reading v1 history.** — a key not currently in the dict emits `DeprecationWarning` naming the miss and the known keys (`_warn_unknown_key`), before falling back to the same `None`/`EMPTY_REGEX` default as before; `.get()` stays silent. Dunder probes (`__deepcopy__`) still raise `AttributeError` outright, and single-underscore probes (`_repr_html_`, IPython's `_ipython_canary_method_should_not_exist_`, etc.) are excluded from the warning too — no real config key starts with `_`, so both guards just keep protocol/introspection probes from misfiring as "typo" warnings. This means internal parser code that reads `self.C.regexes.` unconditionally (e.g. `squash_bidi`'s `bidi`) now warns if a caller's custom `regexes` dict omits that key — a previously-silent partial-override pattern is on the same deprecation path as an actual typo. -**Adding a word to a config set** — first check the *other* sets for the same word (grep `nameparser/config/` or intersect the sets in a `python3 -c`). Real overlaps exist: `do`/`st`/`mc` ∈ `PREFIXES` ∩ `TITLES`/`SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PREFIXES` ∩ `bound_first_names` (position-dependent: leading token → first-name join, mid-name → last-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; dropping `abd` from `bound_first_names`). +**Adding a word to a config set** — first check the *other* sets for the same word (grep `nameparser/config/` or intersect the sets in a `python3 -c`). Real overlaps exist: `do`/`st`/`mc` ∈ `PARTICLES` ∩ `TITLES`/`SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → first-name join, mid-name → last-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; dropping `abd` from `BOUND_GIVEN_NAMES`). -**Before adding a short/common word to `PREFIXES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191). +**Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191). -**Adding a curated sub-set of an existing config set** (must stay ⊆ its parent, e.g. `FIRST_NAME_TITLES` ⊂ `TITLES`) — define the parent as a *static* union in the config module: `TITLES = FIRST_NAME_TITLES | set([...])`. The sub-set is a `_SetManagerAttribute` on `Constants` (like `first_name_titles`), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. The union is import-time only: a runtime `.add()` to the sub-set does **not** propagate to the parent's `SetManager` (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Pin the relationships with import-time `assert`s in the config module itself (see the bottom of `prefixes.py`): `subset ⊆ parent`, and `∩ == ∅` with any set it logically can't overlap (a sub-set member that's also in `TITLES` is silently inert — title handling consumes it first). A violated assert fails at import — before any test runs — so don't also duplicate them as tests. Do **not** assert `titles ∩ prefixes == ∅` — that overlap is intentional (`st`, `do`). +**Adding a curated sub-set of an existing config set** (must stay ⊆ its parent, e.g. `GIVEN_NAME_TITLES` ⊂ `TITLES`) — define the parent as a *static* union in the config module: `TITLES = GIVEN_NAME_TITLES | {...}`. Keep the `frozenset` on the LEFT: `|` takes the left operand's type, so flipping the operands silently yields a plain mutable set again and undoes the 2.2 freeze for that constant. The sub-set is a `_SetManagerAttribute` on `Constants` (like `first_name_titles` — the v1 ATTRIBUTE names did not move in #293, only the module constants did), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. The union is import-time only: a runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Pin the relationships with import-time `assert`s in the config module itself (see the bottom of `particles.py`): `subset ⊆ parent`, and `∩ == ∅` with any set it logically can't overlap (a sub-set member that's also in `TITLES` is silently inert — title handling consumes it first). A violated assert fails at import — before any test runs — so don't also duplicate them as tests. Do **not** assert `titles ∩ prefixes == ∅` — that overlap is intentional (`st`, `do`). **Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`. @@ -221,6 +221,8 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Parsing must never write into `Constants`** — parse-time derived recognition (dotted abbreviations like "Lt.Gov.", conjunction-joined titles/prefixes like "Mr. and Mrs." / "von und zu") lives in per-instance `HumanName._derived_titles` / `_derived_suffixes` / `_derived_conjunctions` / `_derived_prefixes` sets, consulted by `is_title`/`is_suffix`/`is_conjunction`/`is_prefix`/`is_rootname`, reset at the start of `parse_full_name()`, and backfilled in `__setstate__` for pre-existing pickles. Never `self.C..add()` during parsing — `self.C` is usually the shared `CONSTANTS` singleton, so a write makes parse results order-dependent and thread-unsafe (this was a real bug through 1.2.x). `ParsingDoesNotMutateConfigTests` (`tests/test_constants.py`) enforces the invariant by snapshotting the whole config around a parse; it discovers collections structurally via `Constants.__getstate__()`, so new `Constants` collections are watched automatically — nothing to register. If a new derived category is ever needed: add a `_derived_*` set in `__init__`, reset it in `parse_full_name()`, backfill it in `__setstate__`, consult it in the matching `is_*` predicate (store `lc()`-normalized values, mirroring `SetManager`), and add a leak test with a name that triggers it. +**The `nameparser/config` vocabulary constants are frozen, and the 1.x names for them are a bridge for CALLERS only** (2.2, #293) — `TITLES.add("dean")` raises `AttributeError` at the line that writes it, so a runtime addition goes on a config OBJECT instead: `c = Constants(); c.titles.add("dean"); HumanName(name, constants=c)` for the v1 API, `Parser(lexicon=Lexicon.default().add(titles={"dean"}))` for the 2.0 one. Both are warning-free; mutating the shared `CONSTANTS` still works but warns. What the freeze buys is measured in `docs/migrate.rst`: `Lexicon.default()` is `functools.cache`d and reads the constants once, a v1 `Constants` copies them at every construction, and the shared `CONSTANTS` is a copy taken at import — so pre-freeze, an edit after the first parse reached only a freshly built `Constants`, an edit before any parse reached `parse()` and a fresh `Constants` but still not the shared singleton, and which of those happened depended on nothing the reader could see. Separately, every retired 1.x name (`PREFIXES`, `NON_FIRST_NAME_PREFIXES`, `BOUND_FIRST_NAMES`, `FIRST_NAME_TITLES`, `SUFFIX_NOT_ACRONYMS`) still resolves for a caller, with a `DeprecationWarning` naming its new path — but nothing inside `nameparser/` may spell one, in code OR in a comment: `tests/v2/test_config_aliases.py::test_no_internal_code_reads_a_retired_vocabulary_name` scans every `.py` in the package and fails on a hit outside the alias table that owns it. An internal read would also consume the once-per-process warning and leave the real caller told nothing. That scan does not reach `docs/` or this file, which is why the rename needed a prose sweep of its own. + **`HumanName.C` is a property backed by `_C`, but pickles under the public key `'C'`** — `__init__`/direct assignment route through the `C` setter, which calls the shared `_validate_constants` staticmethod (also used by `__init__`) so an invalid value raises `TypeError` immediately instead of surfacing later as an unrelated `AttributeError` deep in parsing (#239). `__getstate__`/`__setstate__` deliberately translate `self._C` ↔ a `'C'` key in the pickled dict (with the usual `CONSTANTS`-singleton-becomes-`None` sentinel) rather than pickling `_C` directly, so the on-disk pickle format hasn't changed across this fix — don't "simplify" that translation away or old pickles/tests that hand-build a state dict with a `'C'` key will break. **Titles permanently shadow first names — be conservative** — any word in `TITLES` is always consumed as a title and can never be parsed as a first name. `"Dean"` is the canonical example: it's a common academic title *and* a common given name, so it is intentionally absent from the default titles (see `docs/customize.rst` — users who need it add it via opt-in `Constants`). Before adding a word to `TITLES`, ask: "Could this plausibly be someone's given name in any culture?" If yes, don't add it globally; it belongs in caller-supplied `Constants` instead. This same caution applies to international honorifics — `Prince`, `Sheikh`, `Frau` are all first names in some contexts. It also applies to any prefix sub-set gated on "never a first name": obscure-looking foreign particles are surprisingly often real given names — `Von` (Von Miller), `Vander` (Brazilian, also the Arcane character). When unsure, exclude — a missing member just means that name isn't auto-handled, whereas a wrong member misparses a real person. @@ -235,9 +237,9 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **A comparison that runs both sides in one tree reports 0 differences, which is what success looks like.** `tools/differential/` has been hardened against this — it generates its baseline worker into a temp dir, strips `PYTHONPATH` from the child, and aborts unless both the version AND the resolved path check out on each side; its README's three invocation traps are the analysis behind that design, worth reading before changing the harness. **The exposure is ad-hoc comparisons you write yourself**, where the same collapse has a cause the harness cannot disarm for you: **the shell's working directory persists between tool calls**, so a two-tree comparison written as two `cd`s silently runs both halves in whichever tree it landed in. Pin absolute paths, and assert `nameparser.__file__` on both sides the way `compare.py` does. The general rule outlives any particular trap: before believing a null result, prove the harness can report a difference — a clean run and a broken harness are the same output. -**`esq` is in both `SUFFIX_ACRONYMS` and `SUFFIX_NOT_ACRONYMS` on purpose — do not "deduplicate" it** — the two branches normalize differently (see the asymmetry gotcha above): the word test strips only edge periods, the acronym test strips all of them. `Esq` matches only through the word set, `E.S.Q.` only through the acronym set. Removing either membership drops a spelling and, for the acronym one, loses the family name (`"John Smith E.S.Q."` → `family='E.S.Q.'`). Pinned by the `suffix_acronym_multidot_spelling` case-table row. The disjointness that IS asserted is `SUFFIX_ACRONYMS_AMBIGUOUS ∩ SUFFIX_NOT_ACRONYMS`, which is a different claim: `suffix_as_written` ORs the branches, so a word membership bypasses the period gate the ambiguous set exists to impose. +**`esq` is in both `SUFFIX_ACRONYMS` and `SUFFIX_WORDS` on purpose — do not "deduplicate" it, and do not describe it as two spellings** — the two branches normalize differently (see the asymmetry gotcha above): the word test strips only edge periods, the acronym test strips all of them. The load-bearing membership is the ACRONYM one, and it carries *both* spellings: `"Esq"` also survives `.replace(".","")` unchanged, so it is in `SUFFIX_ACRONYMS` too. The word membership is therefore inert as shipped — **provably**, not just on a sample: the intersection of the two sets is exactly `{esq}`, no `SUFFIX_WORDS` entry carries an interior period, and `suffixes.py` asserts `SUFFIX_ACRONYMS_AMBIGUOUS ∩ SUFFIX_WORDS == ∅`, so for a word in both, the acronym branch fires wherever the word branch does. Measured to match: `SUFFIX_WORDS − {esq}` changes **no** parse on either API, while `SUFFIX_ACRONYMS − {esq}` changes many and loses the family name on the multi-dot form (`"John Smith E.S.Q."` → `family='E.S.Q.'`). Deliberately no changed-parse COUNT here — three people built three "7 frames × 9 spellings" grids and got three different numbers (12, 15, 18); the count is a property of the grid, the zero and the direction are properties of the code. The word membership is still not junk: it is v1 data parity, and it is what keeps `"Esq"` matching for a caller who removes `esq` from `SUFFIX_ACRONYMS` themselves (verified — after `C.suffix_acronyms.remove('esq')`, `"John Smith Esq"` still parses `suffix='Esq'`, and dropping both memberships gives `family='Esq'`). Pinned by the `suffix_acronym_multidot_spelling` case-table row. The disjointness that IS asserted is `SUFFIX_ACRONYMS_AMBIGUOUS ∩ SUFFIX_WORDS`, which is a different claim: `suffix_as_written` ORs the branches, so a word membership bypasses the period gate the ambiguous set exists to impose. `suffixes.py`'s `# NOT asserted:` block states the same reasoning at the code — keep the two in step. -**`Lexicon.given_name_titles` is deliberately unvalidated against `titles` — do not add a check** — the lookup key is the space-joined run of `Role.TITLE` tokens, built by the parse, and a conjunction inside a run is itself tagged `Role.TITLE`, so `"sir and dame"` is a matchable key whose middle word lives in `conjunctions`. A whole-entry check rejected multi-word entries; a per-word check rejected that one. No static relation over the vocabulary sets decides reachability. An unreachable entry is inert — nothing consults it and nothing misparses — so the condition being "guarded" costs the user nothing while each guard cost a working configuration. If a diagnostic is wanted, it must be non-blocking — the multi-word UserWarning in `_normset` is the shipped example; a raise remains wrong. Note the SHIPPED data cannot carry a multi-word given-name title without a spurious warning: `FIRST_NAME_TITLES ⊆ TITLES` puts the entry in `titles` too, which is per-word warned; user-supplied v2 Lexicons are unaffected (`add(given_name_titles=...)` alone is silent). +**`Lexicon.given_name_titles` is deliberately unvalidated against `titles` — do not add a check** — the lookup key is the space-joined run of `Role.TITLE` tokens, built by the parse, and a conjunction inside a run is itself tagged `Role.TITLE`, so `"sir and dame"` is a matchable key whose middle word lives in `conjunctions`. A whole-entry check rejected multi-word entries; a per-word check rejected that one. No static relation over the vocabulary sets decides reachability. An unreachable entry is inert — nothing consults it and nothing misparses — so the condition being "guarded" costs the user nothing while each guard cost a working configuration. If a diagnostic is wanted, it must be non-blocking — the multi-word UserWarning in `_normset` is the shipped example; a raise remains wrong. Note the SHIPPED data cannot carry a multi-word given-name title without a spurious warning: `GIVEN_NAME_TITLES ⊆ TITLES` puts the entry in `titles` too, which is per-word warned; user-supplied v2 Lexicons are unaffected (`add(given_name_titles=...)` alone is silent). **`_normalize` must reach a fixed point** — storage and match-time share the one fold, and `Lexicon.__setstate__` re-validates, so a value that changes on re-normalization changes under its owner. `strip().strip(".")` alone is not idempotent (`'. a .'` → `' a '` → `'a'`). The loop is the fix; keep any new stripping inside it. **Anything built on `_normalize` must converge too** — `_title_key` joins per-word `_normalize` and DROPS words that fold away; keeping the empty slot stored `'lt .'` as `'lt '`, a key match-time can never rebuild (so the entry is silently inert) and `__setstate__` rejects on the next round-trip as "not written by this version". @@ -271,7 +273,7 @@ Don't use the bare `python3 -m doctest .rst` CLI (no `optionflags`) to che **Prefix-join uses value-based `list.index()`** in `join_on_conjunctions` — fragile when a token value repeats (e.g. a trailing title that's also a suffix acronym, or two `van`s); constrain such lookups to start at `i + 1`. See #100. -**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_NOT_ACRONYMS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, both measured 2026-08-01: a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and trailing position has no such rule at all, so a title word there is neither title nor suffix but a NAME part (`"John Smith Prof."` → `family='Prof.'`) — which is what the comma path already disagrees with. Why it is not simply inverted to "vocabulary decides": `TITLES` holds 692 words that are in no suffix set, and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first trailing rule would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one — `King` is a surname, `King.` is not. +**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, both measured 2026-08-01: a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and trailing position has no such rule at all, so a title word there is neither title nor suffix but a NAME part (`"John Smith Prof."` → `family='Prof.'`) — which is what the comma path already disagrees with. Why it is not simply inverted to "vocabulary decides": `TITLES` holds 692 words that are in no suffix set, and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first trailing rule would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one — `King` is a surname, `King.` is not. ### Tests (`tests/`) diff --git a/docs/customize.rst b/docs/customize.rst index 115f2b04..f0a154a3 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -28,6 +28,14 @@ accepts a plain set of lowercase words, keyword by field name (``titles`` above; ``particles``, ``suffix_words``, and the rest work the same way) — see :doc:`modules` for the full field list. +The default word lists themselves — ``TITLES``, ``PARTICLES`` and the +rest of ``nameparser.config`` — are frozen, so a runtime addition +belongs on a :class:`~nameparser.Lexicon` as above, or on a private +``Constants`` if you are still parsing through ``HumanName``. Those +constants were renamed in 2.2 to match the field names used here; the +1.x names still import, with a ``DeprecationWarning``, until 3.0 — see +:doc:`migrate` for the mapping. + Vocabulary entries are matched one word at a time (``given_name_titles`` excepted), so a multi-word entry like ``titles={"grand moff"}`` can never match; the constructor warns when it sees one diff --git a/docs/migrate.rst b/docs/migrate.rst index b24f13b5..0df2d49f 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -222,7 +222,134 @@ fields: - Pair-valued; set it via ``dataclasses.replace(lexicon, capitalization_exceptions={...})``, not ``add()``/``remove()`` -And behavior/render scalars map onto :class:`~nameparser.Policy` (or a +The vocabulary that feeds both columns lives in ``nameparser.config``, +and in 2.2 its module and constant names moved to the vocabulary the +``Lexicon`` column speaks — particles, bound given names, given-name +titles, suffix words. Terminology only; one of the four kept 1.x's +*meaning* while its ``Lexicon`` counterpart marks the opposite set, so +read the caveat under the table before pairing them up. If you import +the default word lists directly — to read one, extend one, or copy one +into your own configuration — four vocabularies moved: + +.. list-table:: + :header-rows: 1 + :widths: 50 50 + + * - 1.x name + - 2.2 name + * - ``nameparser.config.prefixes`` + - :mod:`nameparser.config.particles` + * - ``prefixes.PREFIXES`` + - ``particles.PARTICLES`` + * - ``prefixes.NON_FIRST_NAME_PREFIXES`` + - ``particles.NON_GIVEN_NAME_PARTICLES`` + * - ``nameparser.config.bound_first_names`` + - :mod:`nameparser.config.bound_given_names` + * - ``bound_first_names.BOUND_FIRST_NAMES`` + - ``bound_given_names.BOUND_GIVEN_NAMES`` + * - ``titles.FIRST_NAME_TITLES`` + - ``titles.GIVEN_NAME_TITLES`` + * - ``suffixes.SUFFIX_NOT_ACRONYMS`` + - ``suffixes.SUFFIX_WORDS`` + +The caveat is on the third row. ``NON_GIVEN_NAME_PARTICLES`` is +``NON_FIRST_NAME_PREFIXES`` renamed and nothing else — same members, +same *never a given name* meaning. It is **not** the constant behind +``Lexicon.particles_ambiguous``, which is that field's complement, even +though the two now sound as though they belong together. Pairing this +table's third row with the field-mapping table above and concluding +that ``NON_GIVEN_NAME_PARTICLES`` is what ``particles_ambiguous`` +holds is exactly the inversion the flip warning below exists to +prevent. + +Every row still resolves, and the old names are removed in 3.0. The two +module rows are import paths and nothing more: importing +``nameparser.config.prefixes`` or ``nameparser.config.bound_first_names`` +still works and says nothing, because the modules are now empty shims. +It is reading a *constant* that reports — by attribute access, by +``from ... import``, and by ``from ... import *`` alike. The read emits +a ``DeprecationWarning`` naming the module and constant to move to, +then returns the constant from its new home. + +The warning fires once per line that reads a retired name, not once per +process, so a repeated read of the same import stays quiet while a +second import somewhere else in your code reports for itself. To find +your own uses, raise ``DeprecationWarning`` — which Python hides by +default outside ``__main__``, so an untouched run of a library that +reads these names on import shows nothing:: + + python -W error::DeprecationWarning -c "import yourapp" + +That stops at the first one, with a traceback whose last frame outside +nameparser is the line to edit. Swap ``error`` for ``default`` to print +them all and keep going. + +Only the data layer moved: the ``CONSTANTS`` attribute names in the +field-mapping table above are v1 facade surface and are unaffected, +so ``constants.prefixes``, +``constants.non_first_name_prefixes``, ``constants.bound_first_names``, +``constants.first_name_titles`` and ``constants.suffix_not_acronyms`` +keep their 1.x spelling for as long as the facade exists. + +Every vocabulary *set* in ``nameparser.config`` is also a ``frozenset`` +as of 2.2 — the renamed ones and the rest. Every set, that is; the one +mapping constant is untouched, and there is a note on it below. The +freeze retires one 1.x idiom outright: ``TITLES.add("dean")`` — editing +a default word list in place — now raises ``AttributeError`` at the +line that writes it, rather than changing some parses and not others +some distance away. + +It was never a dependable way to change a default, because the two +config layers read the module constants at different moments. +``Lexicon.default()`` is cached and reads them exactly once, at its +first call; a v1 ``Constants`` copies them at every construction; and +the shared ``CONSTANTS`` singleton is one such copy, taken at import. +An edit landing *after* the first parse therefore reached only a +freshly built ``Constants`` — neither ``parse()``, whose lexicon was +already built, nor the shared ``CONSTANTS``, which predated the edit. +An edit landing *before* any parse reached ``Lexicon.default()``, and +so ``parse()``, and a fresh ``Constants`` — but still never the shared +``CONSTANTS``. Whether an edit reached a given parse thus depended on +which config objects the program had already built, and one program +could hold two disagreeing defaults with nothing to say so. + +``CAPITALIZATION_EXCEPTIONS`` is the constant the freeze left out. It +is a mapping rather than a set, and it is still a plain mutable +``dict`` — ``CAPITALIZATION_EXCEPTIONS["phd"] = "PhD"`` runs on 2.2 and +raises nothing. Everything just said about split defaults still applies +to it, unchanged and measured on 2.2: an edit after the first parse +reaches a freshly built ``Constants``, and neither +``Lexicon.default()`` nor the shared ``CONSTANTS``. The advice below is +the same advice — configure the object, with +``constants.capitalization_exceptions["phd"] = "PhD"`` on a private +``Constants``, or ``dataclasses.replace(lexicon, +capitalization_exceptions={...})`` for the 2.0 API. + +Configure the objects instead, which both APIs have always supported +and neither the freeze nor the rename affects. For ``HumanName``, build +a private ``Constants`` and pass it:: + + from nameparser import HumanName + from nameparser.config import Constants + + constants = Constants() + constants.titles.add("dean") + name = HumanName("Dean Smith", constants=constants) + +For the 2.0 API, extend the default lexicon and hand it to a parser:: + + from nameparser import Lexicon, Parser + + parser = Parser(lexicon=Lexicon.default().add(titles={"dean"})) + name = parser.parse("Dean Smith") + +Mutating the shared ``CONSTANTS`` singleton still works and still +reaches every ``HumanName`` that reads it, but it warns: it is +deprecated along with the rest of the v1 facade and goes away in 3.0. +Prefer a private ``Constants`` in new code. See :doc:`customize` for +the full set of knobs on each. + +Behavior and render scalars map onto :class:`~nameparser.Policy` (or a rendering argument, where the 2.0 equivalent isn't config at all): .. list-table:: @@ -279,7 +406,12 @@ handing the parser a regex. **complementary** sets, not the same set under a new name. ``non_first_name_prefixes`` lists particles that are *never* read as a given name; ``particles_ambiguous`` lists the particles that - *may* be read as one. Translating a customization means flipping + *may* be read as one. The same holds for the config constant behind + it: ``particles.NON_GIVEN_NAME_PARTICLES`` (1.x + ``prefixes.NON_FIRST_NAME_PREFIXES``) marks the never-given set, so + it is the complement of ``particles_ambiguous`` too, however much + the 2.2 names now suggest otherwise. Translating a customization + means flipping the set: ``particles_ambiguous = lexicon.particles - constants.non_first_name_prefixes``. Copying ``non_first_name_prefixes`` straight into ``particles_ambiguous`` diff --git a/docs/modules.rst b/docs/modules.rst index b19b918d..7766651a 100644 --- a/docs/modules.rst +++ b/docs/modules.rst @@ -209,9 +209,9 @@ HumanName.config Defaults :members: .. automodule:: nameparser.config.suffixes :members: -.. automodule:: nameparser.config.prefixes +.. automodule:: nameparser.config.particles :members: -.. automodule:: nameparser.config.bound_first_names +.. automodule:: nameparser.config.bound_given_names :members: .. automodule:: nameparser.config.conjunctions :members: diff --git a/docs/release_log.rst b/docs/release_log.rst index 680e5933..95bf234b 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -1,5 +1,52 @@ Release Log =========== +* 2.2.0 - Unreleased + + nameparser 2.2 finishes the 2.0 rename at the layer it never + reached. The word lists in ``nameparser.config`` were still named + for v1's fields — prefixes, first names — while the + ``Lexicon`` they feed has spoken of particles and given names since + 2.0. They now agree. The lists are also frozen, which retires + editing one in place as a way to change a default and replaces it + with configuring a ``Lexicon`` or a private ``Constants``. + + Nothing moved between vocabularies and no parse changes: over the + 751 names of the differential corpora, every one of the seven + fields is identical to 2.1 through both the 2.0 and the 1.x API. + What breaks is code that *writes* to a default word list, and code + that imports one by its 1.x name has until 3.0. + + **Breaking Changes** + + - Change every vocabulary set in ``nameparser.config`` to a ``frozenset``: ``TITLES``, ``GIVEN_NAME_TITLES``, ``SUFFIX_WORDS``, ``SUFFIX_ACRONYMS``, ``SUFFIX_ACRONYMS_AMBIGUOUS``, ``GLUED_HONORIFICS``, ``PARTICLES``, ``NON_GIVEN_NAME_PARTICLES``, ``BOUND_GIVEN_NAMES``, ``CONJUNCTIONS`` and ``MAIDEN_MARKERS`` (``KOREAN_SURNAMES`` already was one). Editing one in place -- ``TITLES.add("dean")``, the old way of changing a global default -- now raises ``AttributeError: 'frozenset' object has no attribute 'add'`` at the line that writes it. It was never a reliable way to change a default: whether an edit reached a given parse depended on which config objects had already been built, so one program could hold two disagreeing defaults with nothing to say so. To change the defaults for ``HumanName``, build a private ``Constants`` and pass it (``c = Constants(); c.titles.add("dean"); HumanName(name, constants=c)``); mutating the shared ``CONSTANTS`` still works, but warns and goes away in 3.0. For the 2.0 API, build a lexicon and pass it to a parser (``Parser(lexicon=Lexicon.default().add(titles={"dean"}))``). Neither is affected by this change. ``CAPITALIZATION_EXCEPTIONS`` is a mapping, not a set, and is unchanged. See :doc:`migrate` and :doc:`customize` (#293) + + **Deprecations** + + - Rename the four vocabularies whose 1.x names described the fields they feed in v1's words, so the data layer matches the ``Lexicon``: + + .. list-table:: + :header-rows: 1 + :widths: 50 50 + + * - 1.x name + - 2.2 name + * - ``nameparser.config.prefixes`` + - :mod:`nameparser.config.particles` + * - ``prefixes.PREFIXES`` + - ``particles.PARTICLES`` + * - ``prefixes.NON_FIRST_NAME_PREFIXES`` + - ``particles.NON_GIVEN_NAME_PARTICLES`` + * - ``nameparser.config.bound_first_names`` + - :mod:`nameparser.config.bound_given_names` + * - ``bound_first_names.BOUND_FIRST_NAMES`` + - ``bound_given_names.BOUND_GIVEN_NAMES`` + * - ``titles.FIRST_NAME_TITLES`` + - ``titles.GIVEN_NAME_TITLES`` + * - ``suffixes.SUFFIX_NOT_ACRONYMS`` + - ``suffixes.SUFFIX_WORDS`` + + Every row above still resolves and is removed in 3.0. The two module rows are import paths: importing them still works and says nothing, since both modules are now empty shims. Reading a *constant* -- by attribute access, by ``from ... import``, or by ``from ... import *`` -- emits a ``DeprecationWarning`` naming the module and constant to move to, once per line that reads it rather than once per process, so every place you have to edit is reported rather than only whichever one ran first. ``python -W error::DeprecationWarning -c "import yourapp"`` surfaces them; Python hides ``DeprecationWarning`` outside ``__main__``. Two of the four kept their module, so only the constant moved there. ``SUFFIX_NOT_ACRONYMS`` was also inaccurate as well as dated — ``esq`` is in ``SUFFIX_ACRONYMS`` too. The ``CONSTANTS`` attribute names (``prefixes``, ``non_first_name_prefixes``, ``bound_first_names``, ``first_name_titles``, ``suffix_not_acronyms``) are v1 facade surface and are unchanged. See :doc:`migrate` (#293) + * 2.1.0 - August 7, 2026 nameparser 2.1 makes East Asian names work without configuration. diff --git a/docs/usage.rst b/docs/usage.rst index 623fe00d..e938b0c9 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -87,11 +87,11 @@ name to its full vocabulary set: - adjacent suffixes - ``suffix`` - ``John Smith PhD MD`` → ``PhD, MD`` - * - :mod:`Bound given names ` + * - :mod:`Bound given names ` - the following word - ``given`` - ``abdul salam ahmed`` → ``abdul salam`` - * - :mod:`Particles ` + * - :mod:`Particles ` - the following surname - ``family`` - ``Juan de la Vega`` → ``de la Vega`` diff --git a/nameparser/_config_shim.py b/nameparser/_config_shim.py index dbc59885..baec22da 100644 --- a/nameparser/_config_shim.py +++ b/nameparser/_config_shim.py @@ -664,28 +664,28 @@ def _raise_readonly(name: str) -> None: ) -def _default_vocab() -> dict[str, set[str]]: +def _default_vocab() -> dict[str, frozenset[str]]: # v1 data modules stay the single vocabulary source through 2.x # (same rule as Lexicon.default()). - from nameparser.config.bound_first_names import BOUND_FIRST_NAMES + from nameparser.config.bound_given_names import BOUND_GIVEN_NAMES from nameparser.config.conjunctions import CONJUNCTIONS - from nameparser.config.prefixes import ( - NON_FIRST_NAME_PREFIXES, PREFIXES, + from nameparser.config.particles import ( + NON_GIVEN_NAME_PARTICLES, PARTICLES, ) from nameparser.config.suffixes import ( - SUFFIX_ACRONYMS, SUFFIX_ACRONYMS_AMBIGUOUS, SUFFIX_NOT_ACRONYMS, + SUFFIX_ACRONYMS, SUFFIX_ACRONYMS_AMBIGUOUS, SUFFIX_WORDS, ) - from nameparser.config.titles import FIRST_NAME_TITLES, TITLES + from nameparser.config.titles import GIVEN_NAME_TITLES, TITLES return { - "prefixes": PREFIXES, + "prefixes": PARTICLES, "suffix_acronyms": SUFFIX_ACRONYMS, - "suffix_not_acronyms": SUFFIX_NOT_ACRONYMS, + "suffix_not_acronyms": SUFFIX_WORDS, "suffix_acronyms_ambiguous": SUFFIX_ACRONYMS_AMBIGUOUS, "titles": TITLES, - "first_name_titles": FIRST_NAME_TITLES, + "first_name_titles": GIVEN_NAME_TITLES, "conjunctions": CONJUNCTIONS, - "bound_first_names": BOUND_FIRST_NAMES, - "non_first_name_prefixes": NON_FIRST_NAME_PREFIXES, + "bound_first_names": BOUND_GIVEN_NAMES, + "non_first_name_prefixes": NON_GIVEN_NAME_PARTICLES, } @@ -1038,9 +1038,9 @@ def _build_snapshot(self) -> tuple[Lexicon, Policy, _RenderDefaults]: particles=particles, # complement translation: v1 marks the never-given subset; # v2 marks the may-be-given subset. The trailing union keeps - # a config v1 accepted: prefixes.py asserts its own data has - # no word in both non_first_name_prefixes and - # bound_first_names, but nothing stops a caller adding one at + # a config v1 accepted: particles.py asserts its own data has + # no word in both NON_GIVEN_NAME_PARTICLES and + # BOUND_GIVEN_NAMES, but nothing stops a caller adding one at # runtime, and v1 then lets the bound rule win (leading "dos # Santos Silva" parses first="dos Santos"). Treating such a # word as may-be-given reproduces that rather than raising. @@ -1062,23 +1062,19 @@ def _build_snapshot(self) -> tuple[Lexicon, Policy, _RenderDefaults]: bound_given_names=bound, # v1 Constants has no manager for these (#274 is 2.0 # behavior); the data module is the only source - maiden_markers=frozenset(MAIDEN_MARKERS), + maiden_markers=MAIDEN_MARKERS, # likewise no v1 manager: the unspaced-name segmentation # vocabulary is 2.0 behavior (#271), so it rides in the # snapshot only -- v1's Constants surface stays frozen. - # Unwrapped where maiden_markers above is wrapped: this - # module is born frozen (#293), so no wrap surnames=KOREAN_SURNAMES, # likewise no v1 manager: the glued-honorific tail set is # 2.1 behavior (#308), so it rides in the snapshot only. - # Wrapped, unlike surnames above: suffixes.py is still a - # mutable v1 module, not born-frozen like surnames.py - # (#293). Intersect with the word set: Lexicon enforces - # tails <= suffix_words, and v1 semantics are that deleting - # a suffix word turns the behavior off -- a lingering tail - # simply stops mattering, the same rule ambiguous_acronyms - # gets against suffix_acronyms above. - honorific_tails=frozenset(GLUED_HONORIFICS) & suffix_words, + # Intersect with the word set: Lexicon enforces tails <= + # suffix_words, and v1 semantics are that deleting a suffix + # word turns the behavior off -- a lingering tail simply + # stops mattering, the same rule ambiguous_acronyms gets + # against suffix_acronyms above. + honorific_tails=GLUED_HONORIFICS & suffix_words, # TupleManager is dict[str, object] (v1 parity: values were # never statically str-typed); every real entry is a str, # same assumption _DelimiterManager's sentinel lookup makes diff --git a/nameparser/_facade.py b/nameparser/_facade.py index 3917a572..c72ec469 100644 --- a/nameparser/_facade.py +++ b/nameparser/_facade.py @@ -475,7 +475,7 @@ def _split_last(self) -> tuple[list[str], list[str]]: # v1 parser.py _split_last, verbatim: vocabulary lookup at ACCESS # time (so assigned last names split too), with the all-particle # guard (a family name is assumed not to consist entirely of - # particles, e.g. surname "Do" which also appears in PREFIXES) + # particles, e.g. surname "Do" which also appears in PARTICLES) words = " ".join(self.last_list).split() i = 0 while i < len(words) and self._is_particle(words[i]): diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index b61ff32b..61c08453 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -55,7 +55,7 @@ #: nothing needs the acronym half: the shipped tails are CJK #: honorifics, which are words. #: The same relation is asserted a second time in config/suffixes.py, -#: over the raw GLUED_HONORIFICS/SUFFIX_NOT_ACRONYMS constants at +#: over the raw GLUED_HONORIFICS/SUFFIX_WORDS constants at #: import. The two are not redundant in the way they look: that one #: is an `assert`, stripped under `python -O`, while the check here #: raises unconditionally -- so under -O this is what still holds the @@ -313,7 +313,7 @@ class Lexicon: titles: frozenset[str] = frozenset() #: Titles whose single following name reads as a GIVEN name #: ("sheikh", "sister", ...) rather than a family name. Full - #: default list: :data:`~nameparser.config.titles.FIRST_NAME_TITLES`. + #: default list: :data:`~nameparser.config.titles.GIVEN_NAME_TITLES`. given_name_titles: frozenset[str] = frozenset() #: Post-nominal acronym suffixes, matched with or without periods #: ("phd" matches "PhD" and "Ph.D."). Full default list: @@ -321,7 +321,7 @@ class Lexicon: suffix_acronyms: frozenset[str] = frozenset() #: Post-nominal word suffixes ("jr", "esquire", "iii", ...). Full #: default list: - #: :data:`~nameparser.config.suffixes.SUFFIX_NOT_ACRONYMS`. + #: :data:`~nameparser.config.suffixes.SUFFIX_WORDS`. suffix_words: frozenset[str] = frozenset() #: Subset of suffix_acronyms counted as suffixes only when written #: WITH periods -- their bare forms are common surnames ("ma", @@ -330,14 +330,14 @@ class Lexicon: suffix_acronyms_ambiguous: frozenset[str] = frozenset() #: Family-name particles that chain onto the following piece #: ("van", "de", "bin", ...). Full default list: - #: :data:`~nameparser.config.prefixes.PREFIXES`. + #: :data:`~nameparser.config.particles.PARTICLES`. particles: frozenset[str] = frozenset() #: Subset of particles that can also BE a given name: a leading #: one reads as given and records a particle-or-given ambiguity #: ("Van Johnson", but also "Van Buren"). No constant of its own #: -- the default derives #: as particles minus - #: :data:`~nameparser.config.prefixes.NON_FIRST_NAME_PREFIXES` + #: :data:`~nameparser.config.particles.NON_GIVEN_NAME_PARTICLES` #: (which marks the opposite, never-given subset). particles_ambiguous: frozenset[str] = frozenset() #: Words or characters that join surrounding pieces into one @@ -347,7 +347,7 @@ class Lexicon: #: Given-name prefixes that bind to the following word to form one #: given name ("abdul" -> "Abdul Salam"); never standalone names. #: Full default list: - #: :data:`~nameparser.config.bound_first_names.BOUND_FIRST_NAMES`. + #: :data:`~nameparser.config.bound_given_names.BOUND_GIVEN_NAMES`. bound_given_names: frozenset[str] = frozenset() #: Marker words introducing a birth surname, routed to the maiden #: field ("née", "geb.", "roz.", ...). Full default list: @@ -419,8 +419,8 @@ def __post_init__(self) -> None: # the expensive one -- three working configurations broken # across two attempts. Do not add a third. # - # The v2 form of prefixes.py's NON_FIRST_NAME_PREFIXES-disjoint- - # from-BOUND_FIRST_NAMES assertion. That module guards its own + # The v2 form of particles.py's NON_GIVEN_NAME_PARTICLES-disjoint- + # from-BOUND_GIVEN_NAMES assertion. That module guards its own # data at import; this guards vocabulary a caller supplies. contradictory = ( self.bound_given_names & self.particles) - self.particles_ambiguous @@ -614,39 +614,43 @@ def remove(self, **entries: Iterable[str]) -> Lexicon: @functools.cache def _default_lexicon() -> Lexicon: # v1 data modules are the single source of vocabulary through 2.x. - from nameparser.config.bound_first_names import BOUND_FIRST_NAMES + from nameparser.config.bound_given_names import BOUND_GIVEN_NAMES from nameparser.config.capitalization import CAPITALIZATION_EXCEPTIONS from nameparser.config.conjunctions import CONJUNCTIONS from nameparser.config.maiden_markers import MAIDEN_MARKERS - from nameparser.config.prefixes import NON_FIRST_NAME_PREFIXES, PREFIXES + from nameparser.config.particles import NON_GIVEN_NAME_PARTICLES, PARTICLES from nameparser.config.suffixes import ( GLUED_HONORIFICS, SUFFIX_ACRONYMS, SUFFIX_ACRONYMS_AMBIGUOUS, - SUFFIX_NOT_ACRONYMS, + SUFFIX_WORDS, ) from nameparser.config.surnames import KOREAN_SURNAMES - from nameparser.config.titles import FIRST_NAME_TITLES, TITLES - - # v1 data modules export plain `set[str]`; wrap each at this call site - # so the strictly-typed frozenset[str] fields never see a bare set. + from nameparser.config.titles import GIVEN_NAME_TITLES, TITLES + + # every vocabulary constant is a frozenset since #293, so each one + # feeds its strictly-typed frozenset[str] field as it stands -- and + # this cache reading them ONCE is the reason they are frozen. A + # mutated module set always reached a freshly built Constants, and + # reached this Lexicon only when the edit landed before the first + # call; after it, the cache was already built and the same edit was + # invisible here. Which of the two a program got was not something + # the code doing the mutating could see. # keep in sync with _config_shim.Constants._snapshot() (pinned by the # default-Constants equality test in tests/v2/test_config_shim.py) return Lexicon( - titles=frozenset(TITLES), - given_name_titles=frozenset(FIRST_NAME_TITLES), - suffix_acronyms=frozenset(SUFFIX_ACRONYMS), - suffix_words=frozenset(SUFFIX_NOT_ACRONYMS), - suffix_acronyms_ambiguous=frozenset(SUFFIX_ACRONYMS_AMBIGUOUS), - particles=frozenset(PREFIXES), + titles=TITLES, + given_name_titles=GIVEN_NAME_TITLES, + suffix_acronyms=SUFFIX_ACRONYMS, + suffix_words=SUFFIX_WORDS, + suffix_acronyms_ambiguous=SUFFIX_ACRONYMS_AMBIGUOUS, + particles=PARTICLES, # FLIPPED from v1: v1 marks the never-given subset; v2 marks the # may-be-given subset (migration: complement translation). - particles_ambiguous=frozenset(PREFIXES - NON_FIRST_NAME_PREFIXES), - conjunctions=frozenset(CONJUNCTIONS), - bound_given_names=frozenset(BOUND_FIRST_NAMES), - maiden_markers=frozenset(MAIDEN_MARKERS), - # surnames.py is born frozen (#293) -- no call-site wrap needed, - # unlike the v1 modules above (their wraps drop when #293 lands) + particles_ambiguous=PARTICLES - NON_GIVEN_NAME_PARTICLES, + conjunctions=CONJUNCTIONS, + bound_given_names=BOUND_GIVEN_NAMES, + maiden_markers=MAIDEN_MARKERS, surnames=KOREAN_SURNAMES, - honorific_tails=frozenset(GLUED_HONORIFICS), + honorific_tails=GLUED_HONORIFICS, # pass canonical pair-tuples so this strictly-typed call site never # feeds a Mapping to the tuple-annotated field; __post_init__ # still tolerates a Mapping at runtime for interactive use diff --git a/nameparser/config/__init__.py b/nameparser/config/__init__.py index 05dbae09..d4ce2259 100644 --- a/nameparser/config/__init__.py +++ b/nameparser/config/__init__.py @@ -1,7 +1,14 @@ """v1 import-path preservation (migration spec §3): the Constants shim -lives in nameparser._config_shim. The vocabulary data modules in this -package (titles, suffixes, ...) remain the single source through 2.x. -This package is deleted in 3.0. +lives in nameparser._config_shim. + +Two unrelated things share this package. The names re-exported below -- +``Constants``, ``CONSTANTS``, ``SetManager``, ``TupleManager``, +``RegexTupleManager`` -- are v1 compatibility surface, and go with the +rest of the facade in 3.0. The vocabulary data modules beside them +(:mod:`~nameparser.config.titles`, :mod:`~nameparser.config.particles`, +...) are not compatibility surface at all: they are the word lists the +2.0 :class:`~nameparser.Lexicon` is built from, they are named for its +fields since 2.2 (#293), and its documentation cross-references them. ``RegexTupleManager`` is re-exported unchanged from the shim purely for pickle compatibility: a v1.4 ``Constants`` blob's ``regexes`` field was @@ -11,6 +18,16 @@ blob raises ``AttributeError`` looking up the class, not a clean compatibility failure. """ +# Maintainer note, deliberately outside the docstring: the docstring +# above no longer says "this package is deleted in 3.0", which the +# migration spec's §3 list asserts while enumerating only the shim +# names in its parenthetical. Whether the DATA modules keep this +# package as their home in 3.0 or move under the core is an open +# decision, not something to settle in a published docstring -- and it +# now has a consequence, since Lexicon's public field docs +# cross-reference nameparser.config.particles et al. Resolve it when +# 3.0 is planned; until then this docstring claims only what is +# settled, which is that the re-exports below go. from nameparser._config_shim import CONSTANTS as CONSTANTS from nameparser._config_shim import Constants as Constants from nameparser._config_shim import RegexTupleManager as RegexTupleManager diff --git a/nameparser/config/_deprecated.py b/nameparser/config/_deprecated.py new file mode 100644 index 00000000..60b9f90e --- /dev/null +++ b/nameparser/config/_deprecated.py @@ -0,0 +1,133 @@ +"""The 1.x vocabulary names, served from their 2.2 homes. + +The 2.0 API named its concepts for what they are -- particles, bound +given names, given-name titles, suffix words -- while the data modules +kept the 1.x names a little longer. #293 moved all four to match. A 1.x +name resolves to its 2.2 constant, warns at the line that read it, and +names the path to migrate to; the whole layer goes away in 3.0 with the +rest of the v1 facade. + +Two of the four moved module and all: prefixes -> particles and +bound_first_names -> bound_given_names, whose old modules are now +data-free shims that are nothing but a docstring and an alias table. +The other two renamed a constant in place, so titles.py and suffixes.py +carry their alias table at the bottom of the file, beside their data. + +Same PEP 562 hook as nameparser/locales/__init__.py, but deliberately +without that module's write-back: a retired name stays served by +``__getattr__`` for the life of the process, so every read reaches the +warning. Suppressing the repeats is the warnings module's own job, and +it does it per LOCATION -- ``__warningregistry__`` lives in the READING +module's globals and is keyed on (text, category, lineno). That is the +granularity the advice is written at: one line that reads a retired +name is told once however often it runs, and a second line, in that +file or another, is told for itself. Caching the resolved value into +the module globals instead would silence every reader after the first, +and the first is whoever imported earliest -- routinely a dependency, +whose author is not the person who has to edit anything. + +PEP 562 defines the hook for attribute ACCESS and nothing else, which +is why every alias-bearing module also carries an ``__all__`` naming +its retired names: ``from x import *`` reads ``__all__``, or failing +that the module ``__dict__``, and consults ``__getattr__`` in neither +case. Without the list a star import binds no retired name and issues +no diagnostic. See the note at the ``__all__`` in prefixes.py. +""" +from __future__ import annotations + +import importlib +import sys +import warnings +from collections.abc import Callable, Mapping +from typing import Any + +_MESSAGE = ( + "{module}.{old} is deprecated since 2.2 and will be removed in 3.0; " + "use {new_module}.{new} instead." +) + + +def alias_getattr( + module: str, + aliases: Mapping[str, tuple[str, str]], +) -> tuple[Callable[[str], Any], Callable[[], list[str]]]: + """Build the ``__getattr__``/``__dir__`` pair for a module carrying + deprecated vocabulary names. + + ``aliases`` maps each old attribute name to the ``(module, name)`` + it now lives at. Assign the result at module level, in the ``else`` + of a ``TYPE_CHECKING`` guard that declares the same names:: + + if TYPE_CHECKING: + OLD_NAME: frozenset[str] # 1.x alias, removed in 3.0 (#293) + else: + __getattr__, __dir__ = alias_getattr(__name__, {...}) + + (a placeholder rather than a real retired name, for the reason the + ``stacklevel`` comment below gives) + + The guard is load-bearing. mypy honors an assigned module + ``__getattr__`` (PEP 484's convention for one) and thereafter + answers EVERY missing attribute of that module from its return + type, so a bare assignment turns off missing-attribute checking for + the whole module. On titles.py and suffixes.py, which keep their + live constants and are still imported from, that cost real + checking: ``from nameparser.config.titles import TITLE`` type- + checked clean. Keeping the assignment out of the type checker's + view restores it, and the declarations in the other branch type + each retired name as the ``frozenset[str]`` it is rather than + ``Any``. The package ships ``py.typed``, so both reach callers. + Runtime is untouched -- ``TYPE_CHECKING`` is False, so only the + ``else`` ever runs -- and the two branches delete together in 3.0. + + Which leaves the ``Any`` return below typing nothing outside this + module: mypy reads no module ``__getattr__`` for the alias-bearing + modules any more, and does not analyze the ``else`` branch it is + assigned in. It stays ``Any`` as what ``getattr`` itself returns. + """ + + def __getattr__(name: str) -> Any: # noqa: ANN401 + target = aliases.get(name) + if target is None: + raise AttributeError(f"module {module!r} has no attribute {name!r}") + new_module, new_name = target + # resolved BEFORE warning, so a mistyped alias target fails as + # a ModuleNotFoundError or an AttributeError from here rather + # than first advising the reader to move to a path that does + # not exist. + value = getattr(importlib.import_module(new_module), new_name) + warnings.warn( + _MESSAGE.format( + module=module, old=name, new_module=new_module, new=new_name), + DeprecationWarning, + # 2: the frame that touched the name -- for the + # `from nameparser.config.prefixes import ...` form, the + # importing module, which is the place that has to be + # edited. Which name it imports does not matter here, and + # spelling one out would put a retired name in a file that + # serves no single vocabulary (tests/v2/test_config_aliases + # ::test_no_internal_code_reads_a_retired_vocabulary_name) + stacklevel=2, + ) + return value + + def __dir__() -> list[str]: + # UNION, not just the aliases: a module __dir__ REPLACES the + # default listing rather than adding to it, so dropping the + # module's own globals here would take the live constants out + # of tab completion and every getattr-free member scan -- + # autodoc's included. Pinned by test_config_aliases + # ::test_dir_lists_the_live_names_as_well_as_the_retired_ones. + return sorted(set(vars(sys.modules[module])) | set(aliases)) + + # The table itself, reachable without tripping a warning. __all__ is + # hand-written per module (it must stay in SOURCE order for autodoc, + # which this function cannot know), so the two lists are maintained + # separately and a row added to only one of them is the failure + # fc46a9b closed for the other direction: a table row missing from + # __all__ is silently dropped by `from x import *` with no warning + # and no AttributeError. test_config_aliases + # ::test_every_alias_table_row_reaches_star_import cross-checks them. + __getattr__.deprecated_aliases = dict(aliases) # type: ignore[attr-defined] + + return __getattr__, __dir__ diff --git a/nameparser/config/bound_first_names.py b/nameparser/config/bound_first_names.py index 7dd29e33..8310ab54 100644 --- a/nameparser/config/bound_first_names.py +++ b/nameparser/config/bound_first_names.py @@ -1,28 +1,23 @@ -from nameparser.config._invariants import assert_normalized +"""Deprecated alias module: the bound given-name vocabulary moved to +:mod:`nameparser.config.bound_given_names` in 2.2 (#293), where the +constant name matches the :class:`~nameparser.Lexicon` field it feeds. +Reading a name from here warns and returns the constant from its new +home; this module is deleted in 3.0. +""" +from typing import TYPE_CHECKING -#: Bound Arabic given-name prefixes that attach to the following word to form -#: one first name (e.g. "abdul salam" → first name "abdul salam"). They are -#: never standalone names. Join logic runs in the given-name region only, -#: mirroring :py:data:`~nameparser.config.prefixes.PREFIXES` for last names. -BOUND_FIRST_NAMES: set[str] = { - 'abdul', - 'abdel', - 'abdal', - 'abu', - 'abou', - 'umm', +from nameparser.config._deprecated import alias_getattr - # #269 follow-up: the Arabic-script originals of the entries above. - # Script writes "Abdul Rahman" as two words (عبد + الرحمن -- the - # article attaches to the following word), so عبد alone covers the - # abdul/abdel/abdal variants. Both kunya spellings ship, matching - # the أبو/ابو prefix pair. - 'عبد', # "abd" (servant of) -- عبد الرحمن -> given "عبد الرحمن" - 'أبو', # "abu" (father of), hamza spelling - 'ابو', # "abu", hamza-less spelling - 'أم', # "umm" (mother of), hamza spelling - 'ام', # "umm", hamza-less spelling -} +# Declared for the type checker, served by __getattr__ at runtime -- +# see the note in prefixes.py and alias_getattr's docstring. +if TYPE_CHECKING: + BOUND_FIRST_NAMES: frozenset[str] +else: + __getattr__, __dir__ = alias_getattr(__name__, { + "BOUND_FIRST_NAMES": ( + "nameparser.config.bound_given_names", "BOUND_GIVEN_NAMES"), + }) - -assert_normalized("BOUND_FIRST_NAMES", BOUND_FIRST_NAMES) +# Star imports read __all__ and never the module __getattr__ -- see the +# note in prefixes.py for what that cost before this line existed. +__all__ = ["BOUND_FIRST_NAMES"] diff --git a/nameparser/config/bound_given_names.py b/nameparser/config/bound_given_names.py new file mode 100644 index 00000000..f9ca89a6 --- /dev/null +++ b/nameparser/config/bound_given_names.py @@ -0,0 +1,35 @@ +from nameparser.config._invariants import assert_normalized + +#: Bound Arabic given-name prefixes that attach to the following word to +#: form one given name (e.g. "abdul salam smith" → given name "abdul +#: salam"). They are never standalone names. The join is a group-stage +#: rule on the FIRST non-title piece, so it is not about roles -- it +#: fires whatever name_order later assigns. It reserves a piece for what +#: follows: three pieces that are neither title nor suffix in a main +#: segment, which is why two-word "abdul salam" stays given "abdul" plus +#: family "salam"; only two after a family comma, where the family name +#: is already fixed ("salam, abdul rahman" → given "abdul rahman"). +#: Mirrors :py:data:`~nameparser.config.particles.PARTICLES`, which +#: chains onto the piece that follows it. +BOUND_GIVEN_NAMES: frozenset[str] = frozenset({ + 'abdul', + 'abdel', + 'abdal', + 'abu', + 'abou', + 'umm', + + # #269 follow-up: the Arabic-script originals of the entries above. + # Script writes "Abdul Rahman" as two words (عبد + الرحمن -- the + # article attaches to the following word), so عبد alone covers the + # abdul/abdel/abdal variants. Both kunya spellings ship, matching + # the أبو/ابو prefix pair. + 'عبد', # "abd" (servant of) -- عبد الرحمن -> given "عبد الرحمن" + 'أبو', # "abu" (father of), hamza spelling + 'ابو', # "abu", hamza-less spelling + 'أم', # "umm" (mother of), hamza spelling + 'ام', # "umm", hamza-less spelling +}) + + +assert_normalized("BOUND_GIVEN_NAMES", BOUND_GIVEN_NAMES) diff --git a/nameparser/config/conjunctions.py b/nameparser/config/conjunctions.py index 5e606eae..21a1c1ef 100644 --- a/nameparser/config/conjunctions.py +++ b/nameparser/config/conjunctions.py @@ -1,6 +1,6 @@ from nameparser.config._invariants import assert_normalized -CONJUNCTIONS = { +CONJUNCTIONS = frozenset({ '&', 'and', 'et', @@ -26,7 +26,7 @@ # the "john e smith" bug) protects short names (joins # only with enough rootname pieces). 'و', -} +}) """ Pieces that should join to their neighboring pieces, e.g. "and", "y" and "&". "of" and "the" are also include to facilitate joining multiple titles, diff --git a/nameparser/config/maiden_markers.py b/nameparser/config/maiden_markers.py index 60143846..d064333d 100644 --- a/nameparser/config/maiden_markers.py +++ b/nameparser/config/maiden_markers.py @@ -1,6 +1,6 @@ from nameparser.config._invariants import assert_normalized -MAIDEN_MARKERS = { +MAIDEN_MARKERS = frozenset({ 'née', 'né', 'nee', @@ -18,7 +18,7 @@ 'урождённый', 'урожденный', '旧姓', -} +}) """ Marker words that introduce a birth surname, e.g. "Jane Smith née Jones" (#274). French née/né/nee, German geb./geborene, Dutch geboren, diff --git a/nameparser/config/particles.py b/nameparser/config/particles.py new file mode 100644 index 00000000..0f55a575 --- /dev/null +++ b/nameparser/config/particles.py @@ -0,0 +1,158 @@ +from nameparser.config._invariants import assert_normalized +from nameparser.config.bound_given_names import BOUND_GIVEN_NAMES + +#: The sub-set of :py:data:`PARTICLES` that are *never* a standalone given +#: name. Under the default given-first order that means a name *starting* +#: with one of these has no given name -- the whole thing is a surname +#: (e.g. "de Mesnil" -> family name "de Mesnil"). The reading is scoped to +#: the order on purpose: ``Policy(name_order=FAMILY_FIRST)`` parses the +#: same input as family "de", given "Mesnil", because which side of a +#: leading particle the family name sits on is ``name_order``'s question, +#: not this set's. What membership decides under either order is the +#: ambiguity report -- see :py:data:`PARTICLES` below. +#: Curated to exclude anything that can be a given name in some culture +#: (`al`, `van`, `von`, `della`, `di`, `del`, `da`, `vander`, ...) and +#: anything that is also a bound given-name particle (`abu`). When unsure, +#: leave a word out: a missing member just means that name is not +#: auto-fixed, whereas a wrong member misparses a real person. Must stay a +#: subset of :py:data:`PARTICLES` and disjoint from +#: :py:data:`~nameparser.config.bound_given_names.BOUND_GIVEN_NAMES`. +NON_GIVEN_NAME_PARTICLES = frozenset({ + "'t", + 'af', + 'auf', + 'av', + 'bint', + 'de', + "de'", + 'degli', + 'dei', + 'delle', + 'delli', + 'dello', + 'dem', + 'der', + 'dos', + 'het', + 'ibn', + 'op', + 'ter', + 'vd', + 'vom', + 'zu', + + # #269: Arabic native-script patronymic/clan particles. Unlike their + # Latin transliterations, these live in a script namespace with no + # collision against an unrelated Latin given name, so each is judged + # on its own semantics rather than mirrored blindly: + 'بن', # "bin"/"ibn" (son of) -- never a bare given name. Latin + # 'bin' is in PARTICLES but not in this set; that judgment + # is unchanged by adding the Arabic-script form. + 'بنت', # "bint" (daughter of) -- mirrors Latin 'bint' above. + 'ابن', # "ibn" (son of, alternate spelling) -- mirrors Latin + # 'ibn' above. + 'آل', # "aal" (family/clan of, e.g. "Al Saud") -- distinct from + # the excluded definite article "ال" (#269 explicitly + # excludes standalone "ال"); a clan prefix, never a bare + # given name. + + # #269: Hebrew native-script patronymic particles -- same + # reasoning as the Arabic ones above: no Latin-script collision, + # and neither functions as a standalone given name in Hebrew usage. + # Deferred under the collision rule: 'בר' (Aramaic son-of, as in + # Bar-Lev) -- Bar is a common modern Israeli given name, and the + # surname spelling is hyphenated anyway. + 'בן', # "ben" (son of) + 'בת', # "bat" (daughter of) +}) + +# Maintainer note, deliberately a plain comment ABOVE the `#:` run +# rather than inside it: `#:` would publish it into the API reference, +# where it is advice to nobody, and a plain comment placed *within* the +# run splits it -- autodoc then renders only the fragment below the +# split and silently drops everything above it. Frozen by construction +# (#293) -- `frozenset | set` returns a frozenset, the LEFT operand's +# type wins, so keep the frozenset first. Flipped, this silently yields +# a plain set again and unfreezes the constant. +#: Family-name particles: a particle joins to the piece that follows it +#: to make one new piece, and particles chain, e.g. "von der" and +#: "de la". A particle in a non-leading position also pulls the pieces +#: after it into the same one, up to the next particle run or suffix, +#: which is how a multi-word name piece is recognized. Where that piece +#: lands is a later question: in "pennie von bergen wessels MD", "von" +#: joins each following piece until the suffix "MD", giving the family +#: name "von bergen wessels", while the same chaining in "Smith, Juan +#: de la Cruz" gives the middle name "de la Cruz". A leading +#: particle is the exception and chains nothing, since it may be a given +#: name instead. Where the pieces then land is again a later question, +#: and this one is ``name_order``'s: under the default given-first order +#: a leading :py:data:`NON_GIVEN_NAME_PARTICLES` member makes the whole +#: name a family name ("de la Vega"), while a leading particle outside +#: that set is read as the given name ("Van Johnson") -- whereas +#: ``Policy(name_order=FAMILY_FIRST)`` splits both at the leading +#: particle alike ("de la Vega" -> family "de", given "la Vega"; "Van +#: Johnson" -> family "Van", given "Johnson"), which is the same +#: chains-nothing grouping read the other way round. What membership +#: decides under EITHER order is the report: a leading particle outside +#: :py:data:`NON_GIVEN_NAME_PARTICLES` records a particle-or-given +#: ambiguity for the reading not taken, and one inside it records none. +#: +#: Defined as a static union so every :py:data:`NON_GIVEN_NAME_PARTICLES` +#: member is guaranteed to also be a particle (and still join forward), +#: with no drift -- mirroring ``TITLES = GIVEN_NAME_TITLES | {...}`` in +#: :py:mod:`nameparser.config.titles`. +PARTICLES = NON_GIVEN_NAME_PARTICLES | { + 'aan', + 'aen', + 'abu', + 'al', + 'bar', + 'bat', + 'bin', + 'bon', + 'da', + 'dal', + 'del', + 'dela', + 'della', + 'den', + 'di', + 'dí', + 'do', + 'du', + 'freiherr', + 'freiherrin', + 'heer', + 'la', + 'le', + 'mac', + 'mc', + 'san', + 'santa', + 'st', + 'ste', + 'te', + 'tho', + 'thoe', + 'van', + 'vande', + 'vander', + 'vel', + 'von', + + # #269: Arabic "abu" (father of), left ambiguous like its Latin + # transliteration 'abu' above (both spellings): "Abu Bakr" reads + # "Abu" as a given name, so this stays a PARTICLES-only member, not + # NON_GIVEN_NAME_PARTICLES. + 'أبو', + 'ابو', +} + +# Guard the two invariants the docstring above promises, so a future edit that +# breaks them fails at import time instead of silently drifting until a test +# happens to catch it. +assert NON_GIVEN_NAME_PARTICLES <= PARTICLES, \ + "NON_GIVEN_NAME_PARTICLES must stay a subset of PARTICLES" +assert not (NON_GIVEN_NAME_PARTICLES & BOUND_GIVEN_NAMES), \ + "NON_GIVEN_NAME_PARTICLES must stay disjoint from BOUND_GIVEN_NAMES" +assert_normalized("PARTICLES", PARTICLES) diff --git a/nameparser/config/prefixes.py b/nameparser/config/prefixes.py index f7a89b30..c2b30a82 100644 --- a/nameparser/config/prefixes.py +++ b/nameparser/config/prefixes.py @@ -1,130 +1,38 @@ -from nameparser.config._invariants import assert_normalized -from nameparser.config.bound_first_names import BOUND_FIRST_NAMES +"""Deprecated alias module: the particle vocabulary moved to +:mod:`nameparser.config.particles` in 2.2 (#293), where the constant +names match the :class:`~nameparser.Lexicon` fields they feed. Reading +a name from here warns and returns the constant from its new home; this +module is deleted in 3.0. +""" +from typing import TYPE_CHECKING -#: The sub-set of :py:data:`PREFIXES` that are *never* a standalone first name. -#: A name that *starts* with one of these has no first name -- the whole thing -#: is a surname (e.g. "de Mesnil" -> last name "de Mesnil"). Curated to exclude -#: anything that can be a given name in some culture (`al`, `van`, `von`, -#: `della`, `di`, `del`, `da`, `vander`, ...) and anything that is also a first -#: name prefix (`abu`). When unsure, leave a word out: a missing member just -#: means that name is not auto-fixed, whereas a wrong member misparses a real -#: person. Must stay a subset of :py:data:`PREFIXES` and disjoint from -#: :py:data:`~nameparser.config.bound_first_names.BOUND_FIRST_NAMES`. -NON_FIRST_NAME_PREFIXES = { - "'t", - 'af', - 'auf', - 'av', - 'bint', - 'de', - "de'", - 'degli', - 'dei', - 'delle', - 'delli', - 'dello', - 'dem', - 'der', - 'dos', - 'het', - 'ibn', - 'op', - 'ter', - 'vd', - 'vom', - 'zu', +from nameparser.config._deprecated import alias_getattr - # #269: Arabic native-script patronymic/clan particles. Unlike their - # Latin transliterations, these live in a script namespace with no - # collision against an unrelated Latin given name, so each is judged - # on its own semantics rather than mirrored blindly: - 'بن', # "bin"/"ibn" (son of) -- never a bare given name. Latin - # 'bin' is in PREFIXES but not in this set; that judgment - # is unchanged by adding the Arabic-script form. - 'بنت', # "bint" (daughter of) -- mirrors Latin 'bint' above. - 'ابن', # "ibn" (son of, alternate spelling) -- mirrors Latin - # 'ibn' above. - 'آل', # "aal" (family/clan of, e.g. "Al Saud") -- distinct from - # the excluded definite article "ال" (#269 explicitly - # excludes standalone "ال"); a clan prefix, never a bare - # given name. +# Declared for the type checker, served by __getattr__ at runtime; see +# alias_getattr's docstring for why the assignment has to be hidden +# from mypy. Both branches go in 3.0. +if TYPE_CHECKING: + PREFIXES: frozenset[str] + NON_FIRST_NAME_PREFIXES: frozenset[str] +else: + __getattr__, __dir__ = alias_getattr(__name__, { + "PREFIXES": ("nameparser.config.particles", "PARTICLES"), + "NON_FIRST_NAME_PREFIXES": ( + "nameparser.config.particles", "NON_GIVEN_NAME_PARTICLES"), + }) - # #269: Hebrew native-script patronymic particles -- same - # reasoning as the Arabic ones above: no Latin-script collision, - # and neither functions as a standalone given name in Hebrew usage. - # Deferred under the collision rule: 'בר' (Aramaic son-of, as in - # Bar-Lev) -- Bar is a common modern Israeli given name, and the - # surname spelling is hyphenated anyway. - 'בן', # "ben" (son of) - 'בת', # "bat" (daughter of) -} - -#: Name pieces that appear before a last name. Prefixes join to the piece -#: that follows them to make one new piece. They can be chained together, e.g -#: "von der" and "de la". Because they only appear in middle or last names, -#: they also signify that all following name pieces should be in the same name -#: part, for example, "von" will be joined to all following pieces that are not -#: prefixes or suffixes, allowing recognition of double last names when they -#: appear after a prefixes. So in "pennie von bergen wessels MD", "von" will -#: join with all following name pieces until the suffix "MD", resulting in the -#: correct parsing of the last name "von bergen wessels". -#: -#: Defined as a static union so every :py:data:`NON_FIRST_NAME_PREFIXES` member -#: is guaranteed to also be a prefix (and still join forward), with no drift -- -#: mirroring ``TITLES = FIRST_NAME_TITLES | {...}`` in -#: :py:mod:`nameparser.config.titles`. -PREFIXES = NON_FIRST_NAME_PREFIXES | { - 'aan', - 'aen', - 'abu', - 'al', - 'bar', - 'bat', - 'bin', - 'bon', - 'da', - 'dal', - 'del', - 'dela', - 'della', - 'den', - 'di', - 'dí', - 'do', - 'du', - 'freiherr', - 'freiherrin', - 'heer', - 'la', - 'le', - 'mac', - 'mc', - 'san', - 'santa', - 'st', - 'ste', - 'te', - 'tho', - 'thoe', - 'van', - 'vande', - 'vander', - 'vel', - 'von', - - # #269: Arabic "abu" (father of), left ambiguous like its Latin - # transliteration 'abu' above (both spellings): "Abu Bakr" reads - # "Abu" as a given name, so this stays a PREFIXES-only member, not - # NON_FIRST_NAME_PREFIXES. - 'أبو', - 'ابو', -} - -# Guard the two invariants the docstring above promises, so a future edit that -# breaks them fails at import time instead of silently drifting until a test -# happens to catch it. -assert NON_FIRST_NAME_PREFIXES <= PREFIXES, \ - "NON_FIRST_NAME_PREFIXES must stay a subset of PREFIXES" -assert not (NON_FIRST_NAME_PREFIXES & BOUND_FIRST_NAMES), \ - "NON_FIRST_NAME_PREFIXES must stay disjoint from BOUND_FIRST_NAMES" -assert_normalized("PREFIXES", PREFIXES) +# `from nameparser.config.prefixes import *` consults __all__ and NOTHING +# else -- a module __getattr__ is invisible to it (PEP 562 defines the +# hook for attribute access; star imports without __all__ read the +# module's __dict__ directly). Without this, the one 1.x import form the +# bridge did not cover failed in the mode the bridge exists to prevent: +# no warning, no AttributeError, just a NameError later at an unrelated +# line -- plus `alias_getattr` bound into the caller's namespace. Listing +# the retired names here routes each through __getattr__, so a star +# import warns per name exactly as an attribute read does. +# +# No F822 suppression: the retired names are declared above, in the +# TYPE_CHECKING branch, so ruff sees them bound. Deleting that branch +# in 3.0 without deleting this list is then an error rather than a +# silently-suppressed one. +__all__ = ["NON_FIRST_NAME_PREFIXES", "PREFIXES"] diff --git a/nameparser/config/suffixes.py b/nameparser/config/suffixes.py index ff5e4c4d..1e717ce0 100644 --- a/nameparser/config/suffixes.py +++ b/nameparser/config/suffixes.py @@ -1,6 +1,6 @@ from nameparser.config._invariants import assert_normalized -SUFFIX_NOT_ACRONYMS = { +SUFFIX_WORDS = frozenset({ # #269: Cyrillic мл/ст (junior/senior, the jr/sr analogs) deferred # pending the within-script collision vetting the issue asks for; # 'ст' especially is a plausible false-positive risk (many two- @@ -94,19 +94,34 @@ 'さま', # ja the kana spelling of 様 'くん', # ja the kana spelling of 君 'ちゃん', # ja familiar/diminutive -} +}) """ -Post-nominal pieces that are not acronyms. The parser does not remove periods -when matching against these pieces. +Post-nominal suffixes matched as WORDS: the lookup uses the normalized token, +so only EDGE periods come off and interior ones survive -- "Junior." matches +here, "J.u.n.o.r." does not and stays name text ("John J.u.n.o.r." parses a +family name, on both APIs). The example is deliberately not "J.u.n.i.o.r.", +which fails this lookup too and is a suffix anyway: an interior-period token +that no whole-token set claims goes to ``period_joined_vocab``, which splits +it on its periods and, no chunk being a title, calls the whole thing a +suffix if ANY chunk is suffix vocabulary -- and the chunk "i" is the Roman +numeral listed above. +So membership here is not the last word on a dotted token; the sentence is +about this set's lookup alone. :data:`SUFFIX_ACRONYMS` is the set matched +with every period removed, so it alone covers the multi-dot spelling +"E.S.Q." -- and, having no interior period to lose, "Esq" as well. 'esq' +is listed here too (v1 data): inert against the shipped acronym set, since +dropping it changes no parse, but what keeps "Esq" matching for a caller +who removes it from :data:`SUFFIX_ACRONYMS`. That is why the two sets are +deliberately not asserted disjoint -- see the guard block at the bottom. """ -GLUED_HONORIFICS = { +GLUED_HONORIFICS = frozenset({ # #308: the entries above that may also be peeled off the END of a # name token -- 田中さん, 山田太郎様, 김민준씨. A separate set, not - # SUFFIX_NOT_ACRONYMS reused, because the glued position has no - # token boundary to lean on: the vetting question is not "is this - # a name?" but "can this END a name?", and only entries that can + # SUFFIX_WORDS reused, because the glued position has no token + # boundary to lean on: the vetting question is not "is this a + # name?" but "can this END a name?", and only entries that can # never end one belong here. # kana -- name-final never, in any of the four, and the kana/kanji # split is itself a vetting result: くん ships where 君 cannot, @@ -128,13 +143,13 @@ # its Han twin 博士 is not: that collision is Japanese (博士 = # ひろし) and the hangul spelling carries none of it. '씨', '님', '선생님', '교수님', '박사', '박사님', -} +}) """ -The subset of :data:`SUFFIX_NOT_ACRONYMS` a name token may end WITH, peeled -off as its own token before segmentation (#308). Deliberately harsher than -the spaced set, because a glued tail has no writer-drawn token boundary to -lean on -- these entries are recognized in the SPACED position only: +The subset of :data:`SUFFIX_WORDS` a name token may end WITH, peeled off as +its own token before segmentation (#308). Deliberately harsher than the +spaced set, because a glued tail has no writer-drawn token boundary to lean +on -- these entries are recognized in the SPACED position only: * 양, 군 -- 김지양 and 김지군 are given names ending in these syllables, and 양 is a top-tier surname besides. @@ -152,7 +167,7 @@ as address terms, and only their -님 forms ship. """ -SUFFIX_ACRONYMS_AMBIGUOUS = { +SUFFIX_ACRONYMS_AMBIGUOUS = frozenset({ # Suffix acronyms that also commonly work as given-name nicknames on # their own (e.g. "Ed", "JD"). Read only by HumanName.parse_nicknames() # when deciding whether parenthesized/quoted content is a nickname or a @@ -172,7 +187,7 @@ 'ed', 'jd', 'ma', -} +}) """ Acronym suffixes from SUFFIX_ACRONYMS that also plausibly collide with a @@ -180,7 +195,7 @@ standalone exception list consulted only by parse_nicknames(). """ -SUFFIX_ACRONYMS = { +SUFFIX_ACRONYMS = frozenset({ '8-vsb', 'aas', 'aba', @@ -502,10 +517,13 @@ 'emt-p', 'enp', 'erd', - # Also in SUFFIX_NOT_ACRONYMS, and NOT redundant: the word test - # strips only edge periods while the acronym test strips all of - # them, so the multi-dot spelling "E.S.Q." matches only here while - # bare "Esq" matches only there. + # The load-bearing membership: the acronym test strips every + # period, so this entry is the only thing matching the multi-dot + # spelling, and removing it costs the family name ("John Smith + # E.S.Q." -> family='E.S.Q.'). 'esq' is in SUFFIX_WORDS as well, + # which against this set is inert -- "Esq" has no interior period, + # so it matches here too -- but that is not a duplicate to clean + # up: it is what still matches "Esq" if this entry ever goes. 'esq', 'evp', 'faafp', @@ -810,7 +828,7 @@ 'vcp', 'vd', 'vrd', -} +}) """ Post-nominal acronyms. Titles, degrees and other things people stick after their name @@ -822,30 +840,87 @@ # Guard the invariants the docstrings above promise, so a future edit that # breaks them fails at import time instead of silently drifting until a test -# happens to catch it (same rationale as prefixes.py). Note `assert` is +# happens to catch it (same rationale as particles.py). Note `assert` is # stripped under `python -O`; Lexicon re-checks the relationships at # construction, which is what protects a caller's own vocabulary. assert SUFFIX_ACRONYMS_AMBIGUOUS <= SUFFIX_ACRONYMS, \ "SUFFIX_ACRONYMS_AMBIGUOUS must stay a subset of SUFFIX_ACRONYMS" -# NOT asserted: disjointness of SUFFIX_ACRONYMS and SUFFIX_NOT_ACRONYMS. -# The two sets are matched with different normalization -- the word test -# strips only edge periods, the acronym test strips all of them -- so an -# entry in both is covering two spellings, not duplicated. 'esq' matches -# "Esq" only as a word and "E.S.Q." only as an acronym. +# NOT asserted: disjointness of SUFFIX_ACRONYMS and SUFFIX_WORDS. +# The two are matched with different normalization -- the word test strips +# only edge periods, the acronym test strips all of them -- and no +# SUFFIX_WORDS entry carries an interior period, so for a word in both +# sets the acronym branch fires wherever the word branch does (the assert +# just below keeps such a word out of the period-gated ambiguous subset). +# The single overlap, 'esq', is therefore inert as shipped rather than a +# second spelling: SUFFIX_ACRONYMS covers "E.S.Q." AND "Esq", and dropping +# 'esq' from SUFFIX_WORDS changes no parse. It stays because these sets +# are caller-editable -- it is what still matches "Esq" once 'esq' leaves +# SUFFIX_ACRONYMS -- and an inert overlap is not worth an assert that +# would reject a working config. # DO assert that an ambiguous acronym is not also a plain suffix word: # suffix_as_written ORs the two branches, so the word membership would # bypass the period gate the ambiguous set exists to impose. -assert not (SUFFIX_ACRONYMS_AMBIGUOUS & SUFFIX_NOT_ACRONYMS), \ +assert not (SUFFIX_ACRONYMS_AMBIGUOUS & SUFFIX_WORDS), \ "an ambiguous acronym must not also be a suffix word (the word " \ "branch bypasses its period gate): " \ - f"{sorted(SUFFIX_ACRONYMS_AMBIGUOUS & SUFFIX_NOT_ACRONYMS)}" + f"{sorted(SUFFIX_ACRONYMS_AMBIGUOUS & SUFFIX_WORDS)}" # The peel splits its tail off as a TOKEN and suffix classification is # what claims it downstream, so a tail that is not also a suffix word # would split the name and then leave the piece sitting in it. The # reverse direction is deliberately unguarded: a suffix word that is # not a tail is the ordinary case, and an empty tail set is inert # rather than wrong. -assert GLUED_HONORIFICS <= SUFFIX_NOT_ACRONYMS, \ - "GLUED_HONORIFICS must stay a subset of SUFFIX_NOT_ACRONYMS: " \ - f"{sorted(GLUED_HONORIFICS - SUFFIX_NOT_ACRONYMS)}" -assert_normalized("suffix", SUFFIX_ACRONYMS | SUFFIX_NOT_ACRONYMS) +assert GLUED_HONORIFICS <= SUFFIX_WORDS, \ + "GLUED_HONORIFICS must stay a subset of SUFFIX_WORDS: " \ + f"{sorted(GLUED_HONORIFICS - SUFFIX_WORDS)}" +assert_normalized("suffix", SUFFIX_ACRONYMS | SUFFIX_WORDS) + + +# 1.x name, deprecated in 2.2 and removed in 3.0 (#293). The constant +# did not change module, so this aliases a name to one of this module's +# own globals: a module __getattr__ runs only once the body has finished +# and the module is in sys.modules, so the lookup resolves rather than +# recursing. +from typing import TYPE_CHECKING # noqa: E402 + +from nameparser.config._deprecated import alias_getattr # noqa: E402 + +# Declared for the type checker, served by __getattr__ at runtime -- +# the split is what keeps mypy checking this module's LIVE names; see +# the note in titles.py and alias_getattr's docstring. +if TYPE_CHECKING: + SUFFIX_NOT_ACRONYMS: frozenset[str] +else: + __getattr__, __dir__ = alias_getattr(__name__, { + "SUFFIX_NOT_ACRONYMS": ( + "nameparser.config.suffixes", "SUFFIX_WORDS"), + }) + +# Star imports read __all__ and never the module __getattr__ -- see the +# note in prefixes.py. Live constants listed alongside the retired name +# for the same reason titles.py lists its own. +# +# In SOURCE order, not alphabetical: `automodule :members:` follows +# __all__ where a module defines one, so an alphabetical list here would +# silently reorder this module's entries in modules.html. The retired +# name goes last because autodoc does not document it, and so it has no +# position to preserve. Not for want of SEEING it: the module member +# scan walks dir(), which our __dir__ lists the retired name in, and +# then calls safe_getattr on it -- so an html build resolves this name +# and titles.py's retired one alike, emitting a real DeprecationWarning +# for each. (Invisible in a "build succeeded, 0 warnings" line: Sphinx +# warnings and Python warnings are different channels. Wrap +# sphinx.cmd.build.build_main in warnings.catch_warnings to see them.) +# What declines it is the attribute-doc scan: ModuleAnalyzer parses the +# SOURCE and finds no assignment statement for a name served by +# __getattr__, so autodoc computes is_attr=False, and at module level a +# member that is not an attribute and is neither a class nor a callable +# matches no object type at all -- no documenter is chosen and the +# member is skipped. +__all__ = [ + "SUFFIX_WORDS", + "GLUED_HONORIFICS", + "SUFFIX_ACRONYMS_AMBIGUOUS", + "SUFFIX_ACRONYMS", + "SUFFIX_NOT_ACRONYMS", +] diff --git a/nameparser/config/surnames.py b/nameparser/config/surnames.py index b9e0130c..c84cbd0e 100644 --- a/nameparser/config/surnames.py +++ b/nameparser/config/surnames.py @@ -1,9 +1,8 @@ from nameparser.config._invariants import assert_normalized -# Born a frozenset (#293's convention: new modules have no users to -# bridge, and a mutable module constant would silently desync the -# cached ``Lexicon.default()`` from the shim's per-construction -# copies). +# Born a frozenset (#293's convention: a mutable module constant would +# silently desync the cached ``Lexicon.default()`` from the shim's +# per-construction copies). # # Single-syllable surnames in census rank order, 10 per row; the cut is # the top ~94 -- append new entries at the tail, do not alphabetize diff --git a/nameparser/config/titles.py b/nameparser/config/titles.py index 726d69b5..4b9fe674 100644 --- a/nameparser/config/titles.py +++ b/nameparser/config/titles.py @@ -1,6 +1,6 @@ from nameparser.config._invariants import assert_normalized -FIRST_NAME_TITLES = { +GIVEN_NAME_TITLES = frozenset({ 'aunt', 'auntie', 'brother', @@ -48,17 +48,25 @@ 'الحاجة', # hajj honorific (f) 'الشيخة', # female counterpart of الشيخ 'مهندس', # engineer (a genuine title in Egyptian usage) -} +}) """ -When these titles appear with a single other name, that name is a first name, e.g. +When these titles appear with a single other name, that name is a given name, e.g. "Sir John", "Sister Mary", "Queen Elizabeth". """ -#: **Cannot include things that could also be first names**, e.g. "dean". +# Maintainer note, deliberately a plain comment ABOVE the `#:` run and +# not inside it (a plain comment within a `#:` run splits it, and +# autodoc then drops everything above the split -- see particles.py). +# `#:` would publish this into the API reference, where it is advice to +# nobody. Frozen by +# construction (#293) -- `frozenset | set` returns a frozenset, the LEFT +# operand's type wins, so keep the frozenset first. Flipped, this +# silently yields a plain set again and unfreezes the constant. +#: **Cannot include things that could also be given names**, e.g. "dean". #: Many of these from wikipedia: https://en.wikipedia.org/wiki/Title. -#: The parser recognizes chains of these including conjunctions allowing +#: The parser recognizes chains of these including conjunctions allowing #: recognition titles like "Deputy Secretary of State". -TITLES = FIRST_NAME_TITLES | { +TITLES = GIVEN_NAME_TITLES | { "attaché", "chargé", "d'affaires", @@ -715,7 +723,7 @@ # #269: Cyrillic (ru/uk) -- mr/mrs/dr/prof/academician/pan(i) # honorifics, same title-then-family convention as 'mr'/'dr'/'prof' - # above (not FIRST_NAME_TITLES: "г-н Петров" families the surname + # above (not GIVEN_NAME_TITLES: "г-н Петров" families the surname # just like "Mr. Smith" does). 'г-н', 'г-жа', @@ -777,12 +785,48 @@ # Guard the invariants at import time, so a bad edit fails here instead of -# drifting silently until a test happens to catch it (see prefixes.py). +# drifting silently until a test happens to catch it (see particles.py). # The subset rule holds by construction today -- TITLES is defined as -# FIRST_NAME_TITLES | {...} -- so this pins it against a future edit that -# makes TITLES a standalone set. Lexicon enforces the same rule on -# caller-supplied vocabulary; `assert` is stripped under `python -O`. -assert FIRST_NAME_TITLES <= TITLES, \ - "FIRST_NAME_TITLES must stay a subset of TITLES" -# TITLES covers FIRST_NAME_TITLES, by the subset assert above. +# GIVEN_NAME_TITLES | {...} -- so this pins it against a future edit that +# makes TITLES a standalone set. Note `assert` is stripped under +# `python -O`, and unlike suffixes.py's relations this one has no +# runtime backstop: Lexicon deliberately does NOT validate +# given_name_titles against titles (its "NOT validated" comment gives +# the reasoning), so a caller's +# Lexicon(titles=frozenset({"sir"}), given_name_titles=frozenset({"dame"})) +# is accepted. Under -O nothing checks this relation at all. +assert GIVEN_NAME_TITLES <= TITLES, \ + "GIVEN_NAME_TITLES must stay a subset of TITLES" +# TITLES covers GIVEN_NAME_TITLES, by the subset assert above. assert_normalized("TITLES", TITLES) + + +# 1.x name, deprecated in 2.2 and removed in 3.0 (#293). Unlike the +# module moves in #293, the constant did not change module, so this +# aliases a name to one of this module's own globals: a module +# __getattr__ runs only once the body has finished and the module is in +# sys.modules, so the lookup resolves rather than recursing. +from typing import TYPE_CHECKING # noqa: E402 + +from nameparser.config._deprecated import alias_getattr # noqa: E402 + +# Declared for the type checker, served by __getattr__ at runtime. The +# split is what keeps mypy checking this module's LIVE names: an +# assigned module __getattr__ answers every missing attribute, so a +# plain assignment here made `from ... import TITLE` type-check clean. +# See alias_getattr's docstring. Both branches go in 3.0. +if TYPE_CHECKING: + FIRST_NAME_TITLES: frozenset[str] +else: + __getattr__, __dir__ = alias_getattr(__name__, { + "FIRST_NAME_TITLES": ( + "nameparser.config.titles", "GIVEN_NAME_TITLES"), + }) + +# Star imports read __all__ and never the module __getattr__ -- see the +# note in prefixes.py. This module keeps its live constants, so they are +# listed too: without __all__ a star import bound them and dropped the +# retired name silently; with a PARTIAL __all__ it would bind the +# retired name and drop the live ones instead. +# Source order, not alphabetical -- see the note in suffixes.py. +__all__ = ["GIVEN_NAME_TITLES", "TITLES", "FIRST_NAME_TITLES"] diff --git a/tests/test_bound_first_names.py b/tests/test_bound_given_names.py similarity index 99% rename from tests/test_bound_first_names.py rename to tests/test_bound_given_names.py index 3c1fb475..43bf8605 100644 --- a/tests/test_bound_first_names.py +++ b/tests/test_bound_given_names.py @@ -2,7 +2,7 @@ from tests.base import HumanNameTestBase -class BoundFirstNamesTestCase(HumanNameTestBase): +class BoundGivenNamesTestCase(HumanNameTestBase): # The v1 is_bound_first_name predicate is gone with the other v1 parsing # hooks (#280); the vocabulary's behavior is pinned through the parsing # tests below. diff --git a/tests/test_conjunctions.py b/tests/test_conjunctions.py index 05b27de3..8c7c0b18 100644 --- a/tests/test_conjunctions.py +++ b/tests/test_conjunctions.py @@ -217,7 +217,7 @@ def test_conjunction_in_an_address_with_a_title(self) -> None: def test_conjunction_in_an_address_with_a_first_name_title(self) -> None: hn = HumanName("Her Majesty Queen Elizabeth") self.m(hn.title, "Her Majesty Queen", hn) - # if you want to be technical, Queen is in FIRST_NAME_TITLES + # if you want to be technical, Queen is in GIVEN_NAME_TITLES self.m(hn.first, "Elizabeth", hn) def test_name_is_conjunctions(self) -> None: diff --git a/tests/test_prefixes.py b/tests/test_particles.py similarity index 94% rename from tests/test_prefixes.py rename to tests/test_particles.py index 26b2d75d..3edeb863 100644 --- a/tests/test_prefixes.py +++ b/tests/test_particles.py @@ -2,12 +2,12 @@ from nameparser import HumanName from nameparser.config import CONSTANTS, Constants -from nameparser.config.prefixes import NON_FIRST_NAME_PREFIXES +from nameparser.config.particles import NON_GIVEN_NAME_PARTICLES from tests.base import HumanNameTestBase -class PrefixesTestCase(HumanNameTestBase): +class ParticlesTestCase(HumanNameTestBase): def test_prefix(self) -> None: hn = HumanName("Juan del Sur") @@ -164,22 +164,23 @@ def test_comma_three_conjunctions(self) -> None: self.m(hn.middle, "Q. Xavier", hn) self.m(hn.suffix, "III", hn) - # The subset-of-PREFIXES and disjoint-from-BOUND_FIRST_NAMES invariants - # are enforced by import-time asserts in nameparser/config/prefixes.py, + # The subset-of-PARTICLES and disjoint-from-BOUND_GIVEN_NAMES invariants + # are enforced by import-time asserts in nameparser/config/particles.py, # so they are not repeated as tests here. - def test_non_first_name_prefixes_expected_members(self) -> None: - # 'abu' is in PREFIXES but excluded (it is a bound_first_name); + def test_non_given_name_particles_expected_members(self) -> None: + # 'abu' is in PARTICLES but excluded (it is a bound_first_name); # 'von'/'van'/'della'/'di'/'del' are excluded (they can be first names). - self.assertIn('de', NON_FIRST_NAME_PREFIXES) - self.assertIn('dos', NON_FIRST_NAME_PREFIXES) - self.assertNotIn('abu', NON_FIRST_NAME_PREFIXES) - self.assertNotIn('von', NON_FIRST_NAME_PREFIXES) - self.assertNotIn('van', NON_FIRST_NAME_PREFIXES) - self.assertNotIn('della', NON_FIRST_NAME_PREFIXES) + self.assertIn('de', NON_GIVEN_NAME_PARTICLES) + self.assertIn('dos', NON_GIVEN_NAME_PARTICLES) + self.assertNotIn('abu', NON_GIVEN_NAME_PARTICLES) + self.assertNotIn('von', NON_GIVEN_NAME_PARTICLES) + self.assertNotIn('van', NON_GIVEN_NAME_PARTICLES) + self.assertNotIn('della', NON_GIVEN_NAME_PARTICLES) def test_constants_exposes_non_first_name_prefixes(self) -> None: - self.assertEqual(set(CONSTANTS.non_first_name_prefixes), NON_FIRST_NAME_PREFIXES) + self.assertEqual( + set(CONSTANTS.non_first_name_prefixes), NON_GIVEN_NAME_PARTICLES) def test_non_first_name_prefixes_disjoint_from_titles(self) -> None: # A member that is also a title is consumed as a title before the fold @@ -234,7 +235,7 @@ def test_no_prefix(self) -> None: self.assertEqual(hn.last_prefixes_list, []) def test_do_guard_surname_equals_prefix_word(self) -> None: - # "Do" is in PREFIXES; without the guard last_base would be empty + # "Do" is in PARTICLES; without the guard last_base would be empty hn = HumanName("Anh Do") self.m(hn.last_base, "Do", hn) self.m(hn.last_prefixes, "", hn) diff --git a/tests/test_titles.py b/tests/test_titles.py index 7bfee722..0cf6ce6d 100644 --- a/tests/test_titles.py +++ b/tests/test_titles.py @@ -186,7 +186,7 @@ def test_possible_conflict_with_suffix_that_could_be_initial(self) -> None: self.m(hn.middle, "A.", hn) self.m(hn.suffix, "V, Jr.", hn) - # 'ben' is removed from PREFIXES in v0.2.5 + # 'ben' was removed from the particle set (then PREFIXES) in v0.2.5 # this test could re-enable this test if we decide to support 'ben' as a prefix @pytest.mark.xfail def test_ben_as_conjunction(self) -> None: diff --git a/tests/v2/test_config_aliases.py b/tests/v2/test_config_aliases.py new file mode 100644 index 00000000..4093aacb --- /dev/null +++ b/tests/v2/test_config_aliases.py @@ -0,0 +1,327 @@ +"""The 1.x vocabulary names, served from their 2.2 homes (#293). + +The alias table here is written out literally rather than imported from +the shim modules. Importing their table would make every assertion +below a tautology -- it would prove the bridge is self-consistent, not +that it points where the migration guide says it does. +""" +from __future__ import annotations + +import importlib +import inspect +import pathlib +import warnings + +import pytest + +import nameparser + +#: (old module, old name, new module, new name), one row per alias. +ALIASES = [ + ("nameparser.config.prefixes", "PREFIXES", + "nameparser.config.particles", "PARTICLES"), + ("nameparser.config.prefixes", "NON_FIRST_NAME_PREFIXES", + "nameparser.config.particles", "NON_GIVEN_NAME_PARTICLES"), + ("nameparser.config.bound_first_names", "BOUND_FIRST_NAMES", + "nameparser.config.bound_given_names", "BOUND_GIVEN_NAMES"), + ("nameparser.config.titles", "FIRST_NAME_TITLES", + "nameparser.config.titles", "GIVEN_NAME_TITLES"), + ("nameparser.config.suffixes", "SUFFIX_NOT_ACRONYMS", + "nameparser.config.suffixes", "SUFFIX_WORDS"), +] + + +@pytest.mark.parametrize( + ("old_module", "old_name", "new_module", "new_name"), + ALIASES, + ids=[f"{m.rsplit('.', 1)[-1]}.{n}" for m, n, _, _ in ALIASES], +) +def test_old_name_warns_and_resolves_to_the_new_constant( + old_module: str, old_name: str, new_module: str, new_name: str, +) -> None: + expected = getattr(importlib.import_module(new_module), new_name) + with pytest.warns(DeprecationWarning) as record: + value = getattr(importlib.import_module(old_module), old_name) + assert value is expected + message = str(record[0].message) + # both paths: naming only the destination would let the message + # misidentify which name the caller actually has to edit + assert f"{old_module}.{old_name}" in message, message + assert f"{new_module}.{new_name}" in message, message + assert "3.0" in message, message + + +def test_warning_points_at_the_line_that_read_the_name() -> None: + """A message nobody can trace back to their own code is advice that + cannot be acted on -- #337's scar is exactly this regressing + unnoticed, since a wrong ``stacklevel`` is invisible from inside the + warning call. Only the recorded frame shows it.""" + module = importlib.import_module("nameparser.config.prefixes") + frame = inspect.currentframe() + assert frame is not None + with pytest.warns(DeprecationWarning) as record: + expected_lineno = frame.f_lineno + 1 + module.PREFIXES # noqa: B018 + assert (record[0].filename, record[0].lineno) == (__file__, expected_lineno) + + +def test_from_import_is_attributed_to_the_importing_module() -> None: + """The form the ``stacklevel`` comment singles out, and the one most + callers use. ``from x import Y`` resolves the alias while the + importing module's frame is on top, so the report names the file + holding the import -- the line that has to be edited.""" + code = compile("from nameparser.config.prefixes import PREFIXES\n", + "caller_module.py", "exec") + with pytest.warns(DeprecationWarning) as record: + exec(code, {"__name__": "caller_module"}) + assert (record[0].filename, record[0].lineno) == ("caller_module.py", 1) + + +@pytest.mark.parametrize( + ("old_module", "old_name"), + [(m, n) for m, n, _, _ in ALIASES], + ids=[f"{m.rsplit('.', 1)[-1]}.{n}" for m, n, _, _ in ALIASES], +) +def test_old_name_warns_once_per_read_location( + old_module: str, old_name: str, +) -> None: + """Every line that has to be edited is told, and told once. + + The bridge deliberately does not cache the resolved value back into + the shim module, so the suppression is ``__warningregistry__``'s: + keyed on (text, category, lineno) in the READING module's globals, + it silences a repeat from the same line and lets a new line through. + A write-back would instead hand the single warning to whoever read + first, which in a real program is usually a dependency. + + The filter action is load-bearing, and this test is vacuous under + the wrong one. ``pytest.warns`` installs ``always``, and the suite's + own ``filterwarnings = ["error"]`` raises before recording -- under + either, nothing is ever written to the registry, so all three reads + below report and the assertion measures nothing. ``default`` is the + action that records what it has already shown. Entering and leaving + ``catch_warnings`` bumps the filter version, which invalidates any + registry this file left behind, so each parametrization starts cold + without anyone clearing it. + """ + module = importlib.import_module(old_module) + frame = inspect.currentframe() + assert frame is not None + with warnings.catch_warnings(record=True) as record: + warnings.simplefilter("default") + repeated_lineno = frame.f_lineno + 2 + for _ in range(2): + first = getattr(module, old_name) + fresh_lineno = frame.f_lineno + 1 + second = getattr(module, old_name) + + assert first is second + assert [w.lineno for w in record] == [repeated_lineno, fresh_lineno] + + +@pytest.mark.parametrize( + "old_module", sorted({m for m, _, _, _ in ALIASES})) +def test_unknown_attribute_still_raises(old_module: str) -> None: + module = importlib.import_module(old_module) + with pytest.raises(AttributeError, match="NOT_A_CONSTANT"): + module.NOT_A_CONSTANT # noqa: B018 + + +@pytest.mark.parametrize( + ("old_module", "old_name"), + [(m, n) for m, n, _, _ in ALIASES], + ids=[f"{m.rsplit('.', 1)[-1]}.{n}" for m, n, _, _ in ALIASES], +) +def test_dir_advertises_the_old_names(old_module: str, old_name: str) -> None: + assert old_name in dir(importlib.import_module(old_module)) + + +def test_dir_lists_the_live_names_as_well_as_the_retired_ones() -> None: + """The other half of what these four ``__dir__`` overrides owe. + + A module ``__dir__`` REPLACES the default listing, so an override + that returns only the alias table takes every live constant out of + REPL completion, out of ``inspect.getmembers``, and out of autodoc's + module member scan -- which walks ``dir()`` and would then document + nothing from ``suffixes``/``titles``. The retired-name assertion + above is satisfied by exactly that override, so it has to be said + separately. + + Stated over the whole of ``vars()`` rather than the vocabulary + alone: the union is what the override actually promises, and it + cannot go vacuous the way an empty vocabulary filter can on the two + data-free shim modules. + """ + live_seen = [] + dropped = {} + for old_module in sorted({m for m, _, _, _ in ALIASES}): + module = importlib.import_module(old_module) + missing = set(vars(module)) - set(dir(module)) + if missing: + dropped[old_module] = sorted(missing) + live_seen += [name for name, value in vars(module).items() + if name.isupper() and isinstance(value, frozenset)] + # checked first: the sweep below is equally happy with four modules + # holding no vocabulary at all, which is the shape that would make + # it prove nothing + assert live_seen, ( + "no live vocabulary constant found in any alias-bearing module " + "-- the sweep below would be measuring only dunders") + assert not dropped, ( + f"__dir__ returned less than the module's own globals: {dropped}" + f" -- an override that does not union them hides the live " + f"constants from every getattr-free member scan") + + +@pytest.mark.parametrize( + "old_module", sorted({m for m, _, _, _ in ALIASES})) +def test_every_alias_table_row_reaches_star_import(old_module: str) -> None: + """The direction the star-import test cannot see. + + ``__all__`` is hand-written per module and the alias table is a + second hand-written list. An ``__all__`` entry with no table row + fails loudly (the name resolves to nothing). A table row missing + from ``__all__`` is the silent one: ``from x import *`` reads + ``__all__`` and never the module ``__getattr__``, so the row is + simply dropped -- no warning, no ``AttributeError`` -- which is + verbatim the failure fc46a9b added ``__all__`` to eliminate. The + star-import test derives its expected set from this file's + ``ALIASES``, so it agrees with a truncated ``__all__`` and stays + green. + + ``ALIASES`` itself is checked against the module's table for the + same reason: a row added there and not here would leave every other + assertion in this file blind to the new alias. + """ + module = importlib.import_module(old_module) + table = module.__getattr__.deprecated_aliases # type: ignore[attr-defined] + exported = set(module.__all__) + assert set(table) <= exported, ( + f"{old_module} serves {sorted(set(table) - exported)} through " + f"__getattr__ but omits it from __all__, so `from {old_module} " + f"import *` drops the name silently") + assert set(table) == {n for m, n, _, _ in ALIASES if m == old_module}, ( + f"{old_module}'s alias table and this file's ALIASES disagree; " + f"the literal table here is what proves the bridge points where " + f"the migration guide says, so it has to cover every row") + + +@pytest.mark.parametrize( + "old_module", sorted({m for m, _, _, _ in ALIASES})) +def test_star_import_binds_exactly_the_live_and_retired_names( + old_module: str, +) -> None: + """``from x import *`` consults ``__all__`` and nothing else. + + A module ``__getattr__`` is invisible to it, so before ``__all__`` + landed this was the one 1.x import form the bridge did not cover, + and it failed in the mode the bridge exists to prevent: no warning, + no ``AttributeError``, just a ``NameError`` further down at a line + with nothing to do with the rename -- and ``alias_getattr`` bound + into the caller's namespace in place of the vocabulary. + + The expected set is DERIVED from the module rather than listed, so a + constant added to ``suffixes``/``titles`` without a matching + ``__all__`` entry fails here. A hand-written list would have to be + kept in step by the same person who forgot ``__all__``. + """ + module = importlib.import_module(old_module) + retired = {n for m, n, _, _ in ALIASES if m == old_module} + # A retired name is served by __getattr__ and never written into the + # module, so vars() holds the live constants -- plus whatever else + # the file imported. The type test is what separates the two: + # `TYPE_CHECKING`, imported to hide the __getattr__ assignment from + # mypy, has the name shape of a constant and is not vocabulary. + # Every live constant in these four modules is a frozenset (the 2.2 + # freeze), so a new one still has to reach __all__ or fail here. + live = {n for n, v in vars(module).items() + if n.isupper() and not n.startswith("_") + and isinstance(v, frozenset)} + + namespace: dict[str, object] = {} + with pytest.warns(DeprecationWarning) as record: + exec(f"from {old_module} import *", namespace) # noqa: S102 + + bound = {n for n in namespace if not n.startswith("__")} + assert bound == live | retired + # the helper the bridge is built from is not vocabulary; before + # __all__ it was the only thing a star import bound here + assert "alias_getattr" not in bound + # one warning per retired name, each naming where to go + assert len(record) == len(retired) + for warning in record: + message = str(warning.message) + assert old_module in message, message + assert "3.0" in message, message + assert {n for n in retired + if any(f"{old_module}.{n} " in str(w.message) for w in record) + } == retired + + +#: Serving a 1.x name is an alias table's whole job, so the file +#: holding that table may spell it. Nothing else in the package may, +#: including the bridge machinery itself. One row per retired name, +#: mapped to the package-relative files it is allowed to appear in -- +#: relative paths rather than bare filenames, so a future +#: ``locales/titles.py`` does not inherit ``config/titles.py``'s +#: exemption. +#: +#: The match below is ``name in source``: raw text, not a token, so a +#: mention in a comment or a docstring counts too. That is the intent -- +#: prose naming a retired constant goes stale exactly the way code does +#: -- but it admits two hits that are not stale references, neither of +#: which can hide a real one: +#: +#: * ``NON_FIRST_NAME_PREFIXES`` contains ``PREFIXES``, so a file +#: holding only the longer name trips both rows. It costs a duplicate +#: line in the report; every row is still checked against the file. +#: * an unrelated identifier may simply contain a retired name -- +#: ``TITLE_PREFIXES`` reports as ``PREFIXES``. Renaming is the wrong +#: advice there, so the failure message offers this allow-list as the +#: other remedy. +_RETIRED_NAMES = { + "PREFIXES": ("config/prefixes.py",), + "NON_FIRST_NAME_PREFIXES": ("config/prefixes.py",), + "BOUND_FIRST_NAMES": ("config/bound_first_names.py",), + # these two kept their module; the exemption is for the alias table + # at the bottom of the file, which names them as strings + "FIRST_NAME_TITLES": ("config/titles.py",), + "SUFFIX_NOT_ACRONYMS": ("config/suffixes.py",), +} + + +def test_no_internal_code_reads_a_retired_vocabulary_name() -> None: + """The bridge exists for callers, not for us. + + An internal read of a 1.x name would warn on a path the suite may + never take, so ``filterwarnings = ["error"]`` alone does not pin + this. It would also aim the bridge at the wrong reader: the warning + is attributed to the line that did the read, so a library-internal + one reports a file inside nameparser and hands the caller advice + about code they cannot edit. + """ + package = pathlib.Path(nameparser.__file__).parent + seen = set() + offenders = [] + for path in sorted(package.rglob("*.py")): + source = path.read_text(encoding="utf-8") + relative = path.relative_to(package).as_posix() + for name, allowed in _RETIRED_NAMES.items(): + if name not in source: + continue + seen.add(name) + if relative not in allowed: + offenders.append(f"{relative}: {name}") + # every retired name is spelled in its own allow-listed file, so a + # name the scan never saw at all means the scan is broken rather + # than the tree clean -- the failure mode where this test passes + # while measuring nothing + assert seen == set(_RETIRED_NAMES), ( + f"scanned {package} and never saw " + f"{sorted(set(_RETIRED_NAMES) - seen)}; the alias tables spell " + f"every retired name, so the scan itself is broken") + assert not offenders, ( + "retired 1.x vocabulary names used inside the package; move them " + "to their 2.2 names (#293) -- or, where a hit is an unrelated " + "identifier that merely contains a retired name, add its path to " + f"that row's _RETIRED_NAMES allow-list: {offenders}") diff --git a/tests/v2/test_config_shim.py b/tests/v2/test_config_shim.py index be7d466b..eca72bba 100644 --- a/tests/v2/test_config_shim.py +++ b/tests/v2/test_config_shim.py @@ -461,10 +461,12 @@ def test_bound_never_given_prefix_deviates_on_two_pieces() -> None: def test_snapshot_keeps_a_bound_never_given_prefix_parseable() -> None: - # prefixes.py asserts its own data keeps non_first_name_prefixes - # disjoint from bound_first_names, but nothing stops a v1 caller - # adding one at runtime, and 1.4 accepts it -- letting the bound - # rule win, so "dos Santos Silva" parses first="dos Santos". + # particles.py asserts its own data has no word in both + # NON_GIVEN_NAME_PARTICLES and BOUND_GIVEN_NAMES, so the defaults + # behind non_first_name_prefixes and bound_first_names never + # collide; nothing stops a v1 caller adding one at runtime, and 1.4 + # accepts it -- letting the bound rule win, so "dos Santos Silva" + # parses first="dos Santos". # Lexicon rejects that combination, so the shim promotes such a word # to may-be-given rather than raising on config v1 allowed. c = Constants() diff --git a/tests/v2/test_contracts.py b/tests/v2/test_contracts.py index e2ecc401..375993f3 100644 --- a/tests/v2/test_contracts.py +++ b/tests/v2/test_contracts.py @@ -102,3 +102,140 @@ def test_every_guarded_config_module_is_imported() -> None: f"derivation broke, and an empty roster asserts nothing") for name in guarded: importlib.import_module(f"nameparser.config.{name}") + + +def test_every_vocabulary_constant_is_frozen() -> None: + """A module vocabulary constant must not be mutable (#293). + + ``Lexicon.default()`` is ``functools.cache``d and reads these sets + once, at its first call, while the v1 shim's ``Constants`` copy + from them at every construction. A runtime ``TITLES.add("dean")`` + was therefore always visible to a freshly built ``Constants``, and + visible to the default ``Lexicon`` only when it landed before the + first parse -- after that the cache was already built and the same + edit was invisible there. Two APIs disagreeing about their own + defaults, decided by construction order, and no way from the + mutating code to tell which branch it was on. Frozen makes that + unrepresentable: the mutation raises where it is written. + + It also carries more than it did. ``_default_lexicon()`` used to + wrap every constant in ``frozenset(...)`` on the way into the + ``Lexicon``; #293 dropped the wraps because the sources are frozen, + which makes this test the only thing anywhere that checks they + still are. The import-time ``assert``\\ s in the config modules + check normalization and subset relations, never mutability. + + The roster is DERIVED from the source tree for the same reason the + guarded-module roster above is: a hand-written list fails open on + the next module or the next constant. ``rglob``, not ``glob``, so a + future ``config/`` subpackage is in scope from the day it lands + rather than from the day someone notices. + + Scope stops at ``nameparser/config``, which is where the hazard is: + three consumers read these constants at three different moments + (the cached ``Lexicon.default()``, a per-construction ``Constants``, + the import-time ``CONSTANTS``), so a mutable one lets two defaults + disagree. A locale pack has one consumer and one moment -- the + ``Lexicon(...)`` in its own module body, whose fields are frozen + copies -- so ``locales/zh.py``'s ``_SURNAMES`` could not desync + anything even as a plain ``set``. It is a ``frozenset`` anyway. + + The two deprecated alias modules are in the glob too and contribute + nothing: the bridge deliberately does not write a resolved value + back into their globals (see ``config/_deprecated.py``), so their + ``vars()`` never gains a vocabulary name however often it is read, + and the roster does not depend on what ran first. + """ + import importlib + import pathlib + + import nameparser.config + + # The two mapping constants, EXEMPT and named rather than dropped by + # the isinstance filter without comment. REGEXES is a compiled- + # pattern table rather than vocabulary and was never in #293's + # scope. CAPITALIZATION_EXCEPTIONS is vocabulary-shaped and is a + # decided, in-scope exemption, which means the split-default hazard + # the freeze closes is STILL LIVE for it -- an edit reaches a + # freshly built Constants and neither the cached Lexicon.default() + # nor the shared CONSTANTS. Written down here, in docs/migrate.rst + # and in AGENTS.md so it does not read as covered. + exempt_mappings = { + "capitalization.CAPITALIZATION_EXCEPTIONS", + "regexes.REGEXES", + } + + config_dir = pathlib.Path(nameparser.config.__file__).parent + checked = [] + offenders = [] + exempt_seen = set() + for path in sorted(config_dir.rglob("*.py")): + relative = path.relative_to(config_dir).with_suffix("") + if any(part.startswith("_") for part in relative.parts): + continue + stem = ".".join(relative.parts) + module = importlib.import_module(f"nameparser.config.{stem}") + for name, value in sorted(vars(module).items()): + if not name.isupper(): + continue + qualified = f"{stem}.{name}" + if isinstance(value, dict): + assert qualified in exempt_mappings, ( + f"{qualified} is a mutable mapping constant with no " + f"recorded exemption; freeze it, or add it to " + f"exempt_mappings with the reason it stays mutable") + exempt_seen.add(qualified) + continue + if not isinstance(value, (set, frozenset)): + continue + checked.append(qualified) + if not isinstance(value, frozenset): + offenders.append(qualified) + assert exempt_seen == exempt_mappings, ( + f"exempt_mappings names {sorted(exempt_mappings - exempt_seen)}, " + f"which the sweep never found -- a stale exemption hides the " + f"next mutable mapping that inherits the name") + # A FLOOR, not a presence check: `assert checked` is satisfied by + # one surviving constant, so a filter or a path change that quietly + # dropped twelve of the thirteen would still read as a pass. Twelve + # distinct constants across seven modules, plus the thirteenth entry + # -- particles.BOUND_GIVEN_NAMES, the same object as + # bound_given_names.BOUND_GIVEN_NAMES, imported there for the + # disjointness assert and counted once per module it appears in. + # Raise this when a constant is added; a drop is the regression. + assert len(checked) >= 13, ( + f"only {len(checked)} vocabulary set constants found under " + f"{config_dir} ({checked}) -- the derivation shrank, and a " + f"roster that shrinks silently stops guarding silently") + assert not offenders, ( + f"vocabulary constants must be frozensets (#293): {offenders}") + + +def test_the_documented_replacements_for_an_in_place_edit_work() -> None: + """``docs/migrate.rst``'s two recipes, as behavior (#293). + + Both live there as ``::`` literal blocks, which ``sphinx -b + doctest`` never runs -- so the page that tells a 1.x caller what to + do INSTEAD of ``TITLES.add("dean")`` was the one claim about the + freeze with nothing checking it. "dean" is the canonical example + for this: a common academic title and a common given name, so it is + deliberately absent from the shipped ``TITLES`` and a caller who + wants it has to add it themselves. + """ + from nameparser import HumanName, Lexicon, Parser + from nameparser.config import Constants + from nameparser.config.titles import TITLES + + # what the freeze retired + with pytest.raises(AttributeError): + TITLES.add("dean") # type: ignore[attr-defined] + assert HumanName("Dean Smith").title == "" + + # recipe 1: a private Constants for the v1 API + constants = Constants() + constants.titles.add("dean") + assert HumanName("Dean Smith", constants=constants).title == "Dean" + + # recipe 2: an extended Lexicon for the 2.0 API + parser = Parser(lexicon=Lexicon.default().add(titles={"dean"})) + assert parser.parse("Dean Smith").title == "Dean" diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 56e5aad0..7ad21966 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -44,7 +44,7 @@ class declares, which members an alternation offers. Those are exact from nameparser._lexicon import _normalize from nameparser.config.maiden_markers import MAIDEN_MARKERS from nameparser.config.suffixes import ( - GLUED_HONORIFICS, SUFFIX_ACRONYMS_AMBIGUOUS, SUFFIX_NOT_ACRONYMS) + GLUED_HONORIFICS, SUFFIX_ACRONYMS_AMBIGUOUS, SUFFIX_WORDS) from ._differential_fixtures import ( _CORPUS_NAMES, _LEDGERS, _TOOLS, _UNCLASSIFIED_NAMES, _claimed, _rules, @@ -517,7 +517,7 @@ def test_cjk_corpus_matches_the_case_table() -> None: #: Which vocabulary constant each ledger rule's alternation is a hand #: copy of. A roster rather than an inference: GLUED_HONORIFICS is a -#: SUBSET of SUFFIX_NOT_ACRONYMS (asserted at the bottom of +#: SUBSET of SUFFIX_WORDS (asserted at the bottom of #: nameparser/config/suffixes.py), so "equals one of the two known sets" #: would let a spaced rule that silently narrowed to exactly the glued #: set pass by matching the other member -- a subset check wearing a @@ -527,9 +527,9 @@ def test_cjk_corpus_matches_the_case_table() -> None: #: Keys are matched as substrings of a rule's `issue` and must select #: exactly one entry. The full issue lists are the keys, not a bare #: '#308': both 2.0 rules cite #308 while copying different constants. -_HONORIFIC_SOURCES: dict[str, set[str]] = { - "cjk-honorific-suffix": SUFFIX_NOT_ACRONYMS, # 1.4 - "#307/#308/#320": SUFFIX_NOT_ACRONYMS, # 2.0, spaced +_HONORIFIC_SOURCES: dict[str, frozenset[str]] = { + "cjk-honorific-suffix": SUFFIX_WORDS, # 1.4 + "#307/#308/#320": SUFFIX_WORDS, # 2.0, spaced "#308/#312/#319/#320": GLUED_HONORIFICS, # 2.0, glued } @@ -580,12 +580,12 @@ def _cjk_alternations(name_regex: str) -> list[set[str]]: def test_differential_honorific_rules_match_their_vocabulary() -> None: """The honorific rules' alternations are hand copies of the CJK - entries of SUFFIX_NOT_ACRONYMS (#307) and of GLUED_HONORIFICS - (#308) -- a toml cannot import them. Each expected set is DERIVED - from the config by script membership (a classified codepoint - anywhere in the entry), so adding a CJK honorific without widening - the rule, or widening a rule with something the vocabulary does not - ship, fails here. + entries of SUFFIX_WORDS (#307) and of GLUED_HONORIFICS (#308) -- + a toml cannot import them. Each expected set is DERIVED from the + config by script membership (a classified codepoint anywhere in the + entry), so adding a CJK honorific without widening the rule, or + widening a rule with something the vocabulary does not ship, fails + here. Swept over every ledger and every alternation, because the three copies are anchored three different ways -- a leading '(?:^| )' in @@ -647,7 +647,7 @@ class _LatinCopy(NamedTuple): `vocabulary` is the source of truth, `covers` an audited snapshot of which of its entries the rule's members reach. """ - vocabulary: set[str] + vocabulary: frozenset[str] covers: frozenset[str] @@ -711,7 +711,7 @@ def _unjustified_reach(name_regex: str, members: set[str]) -> list[str]: if not any(pattern.search(name) for pattern in reachable)] -def _reaches_non_vocabulary(member: str, vocabulary: set[str]) -> list[str]: +def _reaches_non_vocabulary(member: str, vocabulary: frozenset[str]) -> list[str]: """Corpus text this member matches that is NOT a vocabulary entry. fullmatch against the vocabulary bounds what a member matches @@ -873,12 +873,12 @@ def test_latin_alternations_mean_something_the_vocabulary_ships() -> None: #: marker in the name. `suffix` and `title` are not like that -- most #: of their diffs come from routing, not from a vocabulary word being #: present -- so adding them would be false rather than strict. -_FIELD_VOCABULARIES: dict[str, set[str]] = { +_FIELD_VOCABULARIES: dict[str, frozenset[str]] = { "maiden": MAIDEN_MARKERS, } -def _carries(name: str, vocabulary: set[str]) -> bool: +def _carries(name: str, vocabulary: frozenset[str]) -> bool: """Whether a name contains a vocabulary entry. Whole-token for ASCII entries, substring for the rest, because a diff --git a/tests/v2/test_lexicon.py b/tests/v2/test_lexicon.py index 08f6aee4..6fce8c0d 100644 --- a/tests/v2/test_lexicon.py +++ b/tests/v2/test_lexicon.py @@ -341,8 +341,8 @@ def test_subset_error_names_the_fix( def test_bound_given_name_that_is_a_particle_must_be_ambiguous() -> None: - # nameparser/config/prefixes.py asserts this on its own data: - # NON_FIRST_NAME_PREFIXES stays disjoint from BOUND_FIRST_NAMES. In + # nameparser/config/particles.py asserts this on its own data: + # NON_GIVEN_NAME_PARTICLES stays disjoint from BOUND_GIVEN_NAMES. In # 2.0's complement model that reads as bound_given_names & particles # <= particles_ambiguous. A particle declared never-to-start-a-given- # name cannot simultaneously bind one; without the check, one of the diff --git a/tests/v2/test_locales.py b/tests/v2/test_locales.py index 9b75b054..269064e1 100644 --- a/tests/v2/test_locales.py +++ b/tests/v2/test_locales.py @@ -1075,7 +1075,7 @@ def test_non_interference_all_packs_combined() -> None: ("أبو مازن", "given", "أبو"), ("أحمد أبو خليل", "family", "أبو خليل"), ("علي ابو خالد", "family", "ابو خالد"), - # "الشيخ" carries the FIRST_NAME_TITLES semantics of its + # "الشيخ" carries the GIVEN_NAME_TITLES semantics of its # transliterated cousin 'sheikh': a single following name reads as # given, not family. ("الشيخ محمد", "given", "محمد"), @@ -1110,7 +1110,7 @@ def test_non_interference_all_packs_combined() -> None: # behavior. ("דוד בן גוריון", "family", "בן גוריון"), ("שרה בת אברהם", "family", "בת אברהם"), - # Hebrew "מר" title (plain title, not FIRST_NAME_TITLES -- like + # Hebrew "מר" title (plain title, not GIVEN_NAME_TITLES -- like # 'mr', the following name reads as family). ("מר דוד לוי", "title", "מר"), # Hebrew title/suffix sweep (#269 follow-up): plain titles (Israeli diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 24137ed0..37d22b23 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -198,8 +198,8 @@ fields = ["suffix", "nickname"] [[change]] issue = "feat(#269) Arabic بن prefix chains onto family (non-Latin new-recognition)" # '‏محمد بن سلمان‏': #269 adds the native-script Arabic -# patronymic particle بن ("bin"/"son of") to PREFIXES/ -# NON_FIRST_NAME_PREFIXES. v1 had no such entry, so it left بن a plain +# patronymic particle بن ("bin"/"son of") to PARTICLES/ +# NON_GIVEN_NAME_PARTICLES. v1 had no such entry, so it left بن a plain # middle-name token ('سلمان' alone as last); 2.0 now chains it onto the # family the same way 'von'/'bin' (Latin) do, giving family 'بن سلمان'. # This is new-recognition on non-Latin input -- the exact behavior @@ -324,7 +324,7 @@ issue = "fix(cjk-honorific-suffix) postnominal honorifics recognized, compoundin # token is a listed honorific -- a mostly-Latin name with one # ('Wang Xiaoming 先生') is inside its shadow, accepted because the # recognized honorific is the diff's cause there too, and the -# alternation is a hand copy of SUFFIX_NOT_ACRONYMS' CJK entries -- +# alternation is a hand copy of SUFFIX_WORDS' CJK entries -- # pinned by tests/v2/test_ledger_guards.py, which derives the expected # set from the config by script membership. Anchored to a WHOLE # trailing token ((?:^| )...$), judged on the NAME STRING: without diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 7b167ce4..75374fda 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -57,7 +57,7 @@ issue = "fix(#271/#272/#298) native-script CJK: family-first order, hangul segme # tests/v2/test_ledger_guards.py pins this copy, and the honorific # alternations below, by sweeping every expected_since_*.toml rather # than naming one (#333). A span removed from the script table, or an -# entry removed from SUFFIX_NOT_ACRONYMS or GLUED_HONORIFICS, now +# entry removed from SUFFIX_WORDS or GLUED_HONORIFICS, now # forces this file to narrow with it instead of leaving a wide twin # behind to classify a real regression as intended. name_regex = "[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65]" @@ -134,7 +134,7 @@ issue = "fix(#307/#308/#320) spaced CJK postnominal honorific routed to suffix" # explain: unanchored, any name ENDING in 양 or 군 would match, and a # real suffix regression on the glued given name '김지양' would be # absorbed as intentional. The alternation is the CJK half of -# SUFFIX_NOT_ACRONYMS. It is written longest-first for readability, +# SUFFIX_WORDS. It is written longest-first for readability, # not for correctness: the trailing `(?=$|[ ,])` forces backtracking # out of a short alternative, so '박사|박사님' matches '박사님' too # (measured 2026-08-05, and the 1.4 ledger's twin lists them in that