From 342cac2689a25e98dfd364542b15a907728fec2b Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 12:52:51 -0700 Subject: [PATCH 1/5] Name the role the leading particle actually took (#355) The PARTICLE_OR_GIVEN emitter in `_assign` hardcoded "read as a given name" into its detail text. Roles are assigned twenty lines above it, from `_effective_order`, so under `Policy(name_order=FAMILY_FIRST)` the head piece is Role.FAMILY and the report described the reading the parse did not take: Parser(policy=Policy(name_order=FAMILY_FIRST)).parse("Van Johnson") -> family='Van', given='Johnson' detail="leading 'Van' may be a family-name particle; read as a given name" `detail` is user-facing -- it is read off `ParsedName.ambiguities` and printed -- so this was a false statement about the parse in hand, not an internal inaccuracy. The fix is the one the same function already applies twelve lines above: the SUFFIX_OR_NAME block computes its "read as X rather than Y" from `token.role.value`, and `AmbiguityKind.SUFFIX_OR_NAME`'s docstring states the principle -- which name part was declined depends on position and `name_order`, so `detail` names it rather than the kind. The particle emitter now reads the role off the same token. The `kind` is deliberately unchanged. The fork genuinely is "particle or given" -- that is what the two readings of "Van Johnson" are, in either order -- and the enum member is public API. Only the human-readable text moved; default-order output is byte-identical. Test lands in tests/v2/pipeline/test_assign.py rather than tests/v2/test_policy.py: the string is the assign stage's output, that file owns the emitter's module and already holds the sibling PARTICLE_OR_GIVEN test, and test_policy.py never runs the pipeline -- it exercises Policy/PolicyPatch construction and validation only. Co-Authored-By: Claude Opus 5 --- docs/release_log.rst | 4 ++++ nameparser/_pipeline/_assign.py | 9 +++++++-- tests/v2/pipeline/test_assign.py | 20 ++++++++++++++++++++ 3 files changed, 31 insertions(+), 2 deletions(-) diff --git a/docs/release_log.rst b/docs/release_log.rst index 95bf234b..e1946a9c 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -20,6 +20,10 @@ Release Log - Change every vocabulary set in ``nameparser.config`` to a ``frozenset``: ``TITLES``, ``GIVEN_NAME_TITLES``, ``SUFFIX_WORDS``, ``SUFFIX_ACRONYMS``, ``SUFFIX_ACRONYMS_AMBIGUOUS``, ``GLUED_HONORIFICS``, ``PARTICLES``, ``NON_GIVEN_NAME_PARTICLES``, ``BOUND_GIVEN_NAMES``, ``CONJUNCTIONS`` and ``MAIDEN_MARKERS`` (``KOREAN_SURNAMES`` already was one). Editing one in place -- ``TITLES.add("dean")``, the old way of changing a global default -- now raises ``AttributeError: 'frozenset' object has no attribute 'add'`` at the line that writes it. It was never a reliable way to change a default: whether an edit reached a given parse depended on which config objects had already been built, so one program could hold two disagreeing defaults with nothing to say so. To change the defaults for ``HumanName``, build a private ``Constants`` and pass it (``c = Constants(); c.titles.add("dean"); HumanName(name, constants=c)``); mutating the shared ``CONSTANTS`` still works, but warns and goes away in 3.0. For the 2.0 API, build a lexicon and pass it to a parser (``Parser(lexicon=Lexicon.default().add(titles={"dean"}))``). Neither is affected by this change. ``CAPITALIZATION_EXCEPTIONS`` is a mapping, not a set, and is unchanged. See :doc:`migrate` and :doc:`customize` (#293) + **Behavior Changes** + + - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, the way ``SUFFIX_OR_NAME`` has always named the part it declined. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355) + **Deprecations** - Rename the four vocabularies whose 1.x names described the fields they feed in v1's words, so the data layer matches the ``Lexicon``: diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 9e63cf9d..6eef369f 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -274,10 +274,15 @@ def _assign_main(seg_idx: int, state: ParseState, head = pieces[name_pieces[0]] if (len(head) == 1 and len(name_pieces) > 1 and "vocab:particle-ambiguous" in tokens[head[0]].tags): + # the loops above gave the head piece its role; which one it + # is follows name_order, so name the role rather than assume + # given -- same reason as SUFFIX_OR_NAME just above. + token = tokens[head[0]] + assert token.role is not None ambiguities.append(PendingAmbiguity( AmbiguityKind.PARTICLE_OR_GIVEN, - f"leading {tokens[head[0]].text!r} may be a family-name " - f"particle; read as a given name", + f"leading {token.text!r} may be a family-name " + f"particle; read as a {token.role.value} name", tuple(head))) diff --git a/tests/v2/pipeline/test_assign.py b/tests/v2/pipeline/test_assign.py index 5c59a8be..f30359d7 100644 --- a/tests/v2/pipeline/test_assign.py +++ b/tests/v2/pipeline/test_assign.py @@ -72,6 +72,26 @@ def test_leading_ambiguous_particle_reads_as_given_with_ambiguity() -> None: assert not _assigned("John Smith").ambiguities +def test_leading_particle_detail_names_the_role_it_took() -> None: + # The fork is the same under either order -- particle or name -- + # but which role the head piece actually took is name_order's + # answer, so the user-facing detail has to read it off the token + # rather than hardcode "given", exactly as SUFFIX_OR_NAME does. + given_first = _assigned("Van Johnson").ambiguities[0] + assert given_first.kind is AmbiguityKind.PARTICLE_OR_GIVEN + assert given_first.detail == ( + "leading 'Van' may be a family-name particle; " + "read as a given name") + family_first = _assigned( + "Van Johnson", Policy(name_order=FAMILY_FIRST)).ambiguities[0] + # kind is public API and stays PARTICLE_OR_GIVEN -- the fork really + # is "particle or given" -- but here the piece was assigned FAMILY. + assert family_first.kind is AmbiguityKind.PARTICLE_OR_GIVEN + assert family_first.detail == ( + "leading 'Van' may be a family-name particle; " + "read as a family name") + + def test_family_comma() -> None: out = _assigned("de la Vega, Juan") assert _by_role(out, Role.FAMILY) == "de la Vega" From c64738167c4e51635e0eabb8f4b55f964904591f Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 12:56:12 -0700 Subject: [PATCH 2/5] Scope the leading-particle docs to the default name order (#355) Six places described a leading particle by naming the FIELD the piece ends up in. Each was written before `Policy(name_order=FAMILY_FIRST)` shipped in 2.1, and each is falsified by it. Measured: default FAMILY_FIRST Van Johnson given='Van' family='Johnson' family='Van' given='Johnson' van Gogh given='van' family='Gogh' family='van' given='Gogh' de la Vega given='' family='de la Vega' family='de' given='la Vega' The rule itself runs in the grouping stage, before roles are assigned, and never consults `policy.name_order` -- so every MECHANISM claim at these sites was already right and is kept verbatim where it stood: a leading particle has no surname to attach to yet and chains nothing, and whether it can double as a given name is what decides the branch. Only the destinations needed scoping, and they are scoped to the default order rather than restated per order. Follows the pattern PR #354 landed in `config/particles.py`: mechanism first, then where the piece lands under the default given-first order, with the order-dependence stated instead of implied. That docstring carries the long version for both orders, so these are deliberately economical -- the two `.rst` prose sites and `concepts.rst` take a clause, and only `_lexicon.py` and `_types.py` spell out what membership decides under EITHER order (the ambiguity report, which is measured identical under both). `_types.py`'s `PARTICLE_OR_GIVEN` now closes the way its sibling `SUFFIX_OR_NAME` three entries up already did -- the declined part depends on `name_order`, so `detail` names it rather than the kind -- which the previous commit made true of the emitter. The sixth site is the paragraph one below the fifth in customize.rst, outside the issue's enumeration but the same defect: "take it out of the ambiguous set and leading `van` becomes part of the surname" is a destination stated unconditionally. It gets the section's one existing caveat rather than a second copy of it, and leads with the effect that does hold under every order. That effect is the ambiguity, not a field -- under FAMILY_FIRST the knob moves nothing else: van LISTED default given='van' family='Gogh' amb=[particle-or-given] van LISTED FAMILY_FIRST given='Gogh' family='van' amb=[particle-or-given] van REMOVED default given='' family='van Gogh' amb=[] van REMOVED FAMILY_FIRST given='Gogh' family='van' amb=[] No doctests were added or changed. The existing ones in usage.rst and customize.rst all use the default `parse()` and remain true; `sphinx -b doctest` is 223 tests, 0 failures, and a fresh `sphinx -b html` build emits no warnings. Per the lean-docs rule in AGENTS.md, the non-default order gets prose and the unit test added in the previous commit, not a demonstration block. Deliberately untouched: `post_rules` rule 1b keys on roles, so the never-given fold that makes "de Mesnil" wholly a surname does not fire under FAMILY_FIRST -- which is also why the bottom two rows above are identical. That asymmetry is a behavior question filed separately, and nothing here asserts what a non-default order does to a never-given particle. Co-Authored-By: Claude Opus 5 --- docs/concepts.rst | 12 +++++++----- docs/customize.rst | 16 +++++++++++----- docs/usage.rst | 8 +++++--- nameparser/_lexicon.py | 13 +++++++++---- nameparser/_types.py | 7 +++++-- 5 files changed, 37 insertions(+), 19 deletions(-) diff --git a/docs/concepts.rst b/docs/concepts.rst index 566fc672..110c02f1 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -159,11 +159,13 @@ Some calls are irreducibly ambiguous — both readings are legitimate, and no amount of rule-tuning resolves them without breaking some other name. Those surface as entries on ``ParsedName.ambiguities`` instead of being silently guessed away. The canonical example: a leading "Van" -reads as a given name — the right call for the actor Van Johnson, the -wrong one for a bare "Van Buren", and nothing in the two-word shape -distinguishes them — so the parse records a ``particle-or-given`` -ambiguity alongside its answer. You can inspect ``ambiguities`` to decide, case -by case, whether your data needs a second look. +reads as a name of its own — the given name, under the default name +order — rather than as the start of a surname, which is the right call +for the actor Van Johnson and the wrong one for a bare "Van Buren", +and nothing in the two-word shape distinguishes them. So the parse +records a ``particle-or-given`` ambiguity alongside its answer. You +can inspect ``ambiguities`` to decide, case by case, whether your data +needs a second look. An ambiguity records a *decision*, not a word. The same token in a different position may present no fork at all: ``do`` is in the diff --git a/docs/customize.rst b/docs/customize.rst index f0a154a3..e2a10816 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -166,10 +166,14 @@ a suffix only when written with periods: 'M.A.' ``particles_ambiguous`` is the same idea for surname particles. A -particle listed there may also be a given name, so a name that starts -with one keeps its given name; a particle *not* listed there is never a -given name, so a name starting with it has no given name at all — the -whole thing is the surname: +particle listed there may also be a given name, which is what makes a +leading one a decision to take; a particle *not* listed there never +is, so there is nothing to decide. Under the default name order that +shows up as whether the name has a given name at all: one that starts +with a listed particle keeps it, while one starting with an unlisted +particle has no given name — the whole thing is the surname. (Which +field each piece lands in is ``name_order``'s question, covered +below.) .. doctest:: @@ -181,7 +185,9 @@ whole thing is the surname: 'de Mesnil' If your data never uses ``Van`` as a given name, take it out of the -ambiguous set and leading ``van`` becomes part of the surname: +ambiguous set: a leading ``van`` is then no decision at all, so no +ambiguity is recorded, and under the default order it becomes part of +the surname: .. doctest:: diff --git a/docs/usage.rst b/docs/usage.rst index e938b0c9..0cf93389 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -116,9 +116,11 @@ names together as easily as two surnames: 'de la Vega y Rodriguez' Position matters in exactly one place: the start of a name. A particle -there has no surname to attach to yet, so it either becomes the given -name or turns the whole name into a surname, depending on whether it is -one that can double as a given name: +there has no surname to attach to yet, so what decides the reading is +whether it is one that can double as a given name. Under the default +name order it either becomes the given name or turns the whole name +into a surname; under another ``name_order`` — see :doc:`customize` — +where the pieces land follows that order instead: .. doctest:: diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index 61c08453..2bde8886 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -332,10 +332,15 @@ class Lexicon: #: ("van", "de", "bin", ...). Full default list: #: :data:`~nameparser.config.particles.PARTICLES`. particles: frozenset[str] = frozenset() - #: Subset of particles that can also BE a given name: a leading - #: one reads as given and records a particle-or-given ambiguity - #: ("Van Johnson", but also "Van Buren"). No constant of its own - #: -- the default derives + #: Subset of particles that can also BE a given name, which is + #: what makes a leading one a fork rather than a plain particle: + #: it stays a name piece of its own instead of chaining onto what + #: follows. Under the default given-first order that piece is the + #: given name ("Van Johnson", but also "Van Buren"); where it + #: lands under another ``name_order`` is that order's question, + #: not this set's. What membership decides under EITHER order is + #: that the fork is recorded as a particle-or-given ambiguity. + #: No constant of its own -- the default derives #: as particles minus #: :data:`~nameparser.config.particles.NON_GIVEN_NAME_PARTICLES` #: (which marks the opposite, never-given subset). diff --git a/nameparser/_types.py b/nameparser/_types.py index a4298a61..75bc7512 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -356,10 +356,13 @@ class AmbiguityKind(StrEnum): #: Smith B"). Which name part was declined depends on position and #: ``name_order``, so ``detail`` names it rather than the kind. SUFFIX_OR_NAME = "suffix-or-name" - #: A leading ambiguous particle was read as a given name -- the + #: A leading ambiguous particle was read as a name piece of its + #: own rather than as a particle chaining onto what follows -- the #: right call for "Van Johnson" (the actor's given name), the #: wrong one for a bare "Van Buren" (the presidential surname); - #: the two-word shape cannot distinguish them. + #: the two-word shape cannot distinguish them. Which name part it + #: was read as depends on ``name_order`` -- the given name under + #: the default -- so ``detail`` names it rather than the kind. PARTICLE_OR_GIVEN = "particle-or-given" #: A nickname/maiden delimiter opened without closing (or closed #: without opening); the text was kept as literal name content, so From e03a0ae46556bb905e0ffff76b8204c9a95ca659 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 13:22:22 -0700 Subject: [PATCH 3/5] Pin the detail to the role, not to name_order (#355) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit stopped hardcoding "given" in the PARTICLE_OR_GIVEN detail, but the test it shipped with only used inputs where the assigned role and `policy.name_order[0]` agree. So an implementation reading `state.policy.name_order[0].value` -- which is the wrong answer, and the exact falsehood #355 was filed about -- passed the whole suite and all 223 doctests. Measured, before this commit, with that substitution applied to a copy of the tree: 3110 passed, 20 skipped, 11 xfailed The two implementations come apart on the script_orders path (#271), which `_effective_order` resolves ahead of `name_order` and which therefore flips the effective order without touching `name_order` at all. With a Han ambiguous particle: parse('毛 泽东') policy.name_order[0] given role actually assigned family `test_leading_particle_detail_follows_the_effective_order` is that case, and it is the only test the substitution now breaks: FAILED tests/v2/pipeline/test_assign.py:: test_leading_particle_detail_follows_the_effective_order 1 failed, 3116 passed, 20 skipped, 11 xfailed `_assign`'s new comment named the same wrong mechanism -- it said the role "follows name_order" nine lines under `order = _effective_order(...)`, writing the wrong implementation into the comment that justifies the right one. It now says where `order` actually comes from. The `_group` half of this kind was unpinned in the other direction: replacing its whole detail string with `f"{tokens[i].text!r} MUTATED DETAIL TEXT "` also passed everything. That emitter's order-invariance is load-bearing -- it is why the leading-particle docs can scope DESTINATIONS to the default order without qualifying this text -- so `test_chained_particle_detail_is_order_invariant` asserts the exact string under all three orders. "Dr. Van Johnson" takes the chained branch under every one of them (family 'Van Johnson', identical detail), so the assertion is the same string three times, which is the point. Under the mutation, 3 failed. `test_leading_particle_detail_names_the_role_it_took` is parametrized over FAMILY_FIRST_GIVEN_LAST as well, and the facade-level twin sits beside its exact sibling `test_ambiguous_acronym_detail_names_the_role _it_got` -- `detail` is public output, and nothing checked that the role it names survives assembly into `ParsedName`. Tuple-unpacking (`(amb,) = ...`) replaces `.ambiguities[0]` throughout, matching test_parser.py's existing style and pinning the count once. No behavior change: tests and one comment only. pytest 3117 passed, 20 skipped, 11 xfailed mypy Success: no issues found in 103 source files ruff All checks passed! sphinx -b html build succeeded sphinx -b doctest 223 tests, 0 failures Co-Authored-By: Claude Opus 5 --- nameparser/_pipeline/_assign.py | 9 +++-- tests/v2/pipeline/test_assign.py | 58 ++++++++++++++++++++++---------- tests/v2/test_parser.py | 41 ++++++++++++++++++++++ 3 files changed, 88 insertions(+), 20 deletions(-) diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 6eef369f..3975af76 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -274,9 +274,12 @@ def _assign_main(seg_idx: int, state: ParseState, head = pieces[name_pieces[0]] if (len(head) == 1 and len(name_pieces) > 1 and "vocab:particle-ambiguous" in tokens[head[0]].tags): - # the loops above gave the head piece its role; which one it - # is follows name_order, so name the role rather than assume - # given -- same reason as SUFFIX_OR_NAME just above. + # the loops above gave the head piece its role from + # `order`, which is _effective_order's answer and not + # necessarily name_order's -- a script_orders entry + # overrides it. So read the role off the token rather than + # assume given, or re-derive it here; same reason as + # SUFFIX_OR_NAME just above. token = tokens[head[0]] assert token.role is not None ambiguities.append(PendingAmbiguity( diff --git a/tests/v2/pipeline/test_assign.py b/tests/v2/pipeline/test_assign.py index f30359d7..15da5347 100644 --- a/tests/v2/pipeline/test_assign.py +++ b/tests/v2/pipeline/test_assign.py @@ -1,4 +1,6 @@ # tests/v2/pipeline/test_assign.py +import pytest + from nameparser._lexicon import Lexicon from nameparser._pipeline._assign import assign from nameparser._pipeline._classify import classify @@ -24,8 +26,9 @@ ) -def _assigned(text: str, policy: Policy | None = None) -> ParseState: - state = ParseState(original=text, lexicon=_LEX, +def _assigned(text: str, policy: Policy | None = None, + lexicon: Lexicon | None = None) -> ParseState: + state = ParseState(original=text, lexicon=lexicon or _LEX, policy=policy or Policy()) return assign(group(classify(segment(tokenize( extract_delimited(state)))))) @@ -72,23 +75,44 @@ def test_leading_ambiguous_particle_reads_as_given_with_ambiguity() -> None: assert not _assigned("John Smith").ambiguities -def test_leading_particle_detail_names_the_role_it_took() -> None: - # The fork is the same under either order -- particle or name -- - # but which role the head piece actually took is name_order's +@pytest.mark.parametrize("policy,role", [ + (None, "given"), + (Policy(name_order=FAMILY_FIRST), "family"), + (Policy(name_order=FAMILY_FIRST_GIVEN_LAST), "family"), +]) +def test_leading_particle_detail_names_the_role_it_took( + policy: Policy | None, role: str) -> None: + # The fork is the same under every order -- particle or name -- + # but which role the head piece actually took is the assignment's # answer, so the user-facing detail has to read it off the token # rather than hardcode "given", exactly as SUFFIX_OR_NAME does. - given_first = _assigned("Van Johnson").ambiguities[0] - assert given_first.kind is AmbiguityKind.PARTICLE_OR_GIVEN - assert given_first.detail == ( - "leading 'Van' may be a family-name particle; " - "read as a given name") - family_first = _assigned( - "Van Johnson", Policy(name_order=FAMILY_FIRST)).ambiguities[0] - # kind is public API and stays PARTICLE_OR_GIVEN -- the fork really - # is "particle or given" -- but here the piece was assigned FAMILY. - assert family_first.kind is AmbiguityKind.PARTICLE_OR_GIVEN - assert family_first.detail == ( - "leading 'Van' may be a family-name particle; " + # kind is public API and stays PARTICLE_OR_GIVEN throughout: the + # fork really is "particle or given" even where the piece landed + # in FAMILY. + (amb,) = _assigned("Van Johnson", policy).ambiguities + assert amb.kind is AmbiguityKind.PARTICLE_OR_GIVEN + assert amb.detail == ( + f"leading 'Van' may be a family-name particle; " + f"read as a {role} name") + + +def test_leading_particle_detail_follows_the_effective_order() -> None: + # Reading policy.name_order[0] instead of the token's own role + # would pass every case above, because there the two agree. They + # come apart on the script_orders path (#271): a wholly-Han name + # resolves family-first through _effective_order while name_order + # is untouched and still reads given-first. The head piece is the + # FAMILY name here, and the detail has to say so. + han = _LEX.add(particles={"毛"}, particles_ambiguous={"毛"}) + out = _assigned("毛 泽东", lexicon=han) + assert out.policy.name_order[0] is Role.GIVEN + assert out.policy.script_orders[0][0] is Script.HAN + assert _by_role(out, Role.FAMILY) == "毛" + assert _by_role(out, Role.GIVEN) == "泽东" + (amb,) = out.ambiguities + assert amb.kind is AmbiguityKind.PARTICLE_OR_GIVEN + assert amb.detail == ( + "leading '毛' may be a family-name particle; " "read as a family name") diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 6cfb7cfd..a1ab63af 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -253,6 +253,26 @@ def test_ambiguous_acronym_detail_names_the_role_it_got() -> None: assert "family name" not in n.ambiguities[0].detail +def test_leading_particle_detail_names_the_role_it_got() -> None: + # the same requirement as the acronym above, for the other kind: + # `detail` is public output, so the role it names has to survive + # assembly into ParsedName under a non-default order, not just be + # right where _assign builds it + fam_first = Parser(policy=Policy(name_order=FAMILY_FIRST)) + n = fam_first.parse("Van Johnson") + assert (n.family, n.given) == ("Van", "Johnson") + (amb,) = n.ambiguities + assert amb.kind is AmbiguityKind.PARTICLE_OR_GIVEN + assert amb.detail == ( + "leading 'Van' may be a family-name particle; " + "read as a family name") + # the default order is untouched by that change + (default,) = parse("Van Johnson").ambiguities + assert default.detail == ( + "leading 'Van' may be a family-name particle; " + "read as a given name") + + def test_trailing_roman_numeral_reports_the_fork() -> None: # a trailing single letter is a name part unless it happens to be a # roman numeral, in which case it is silently reclassified -- and @@ -318,6 +338,27 @@ def test_chained_particle_detail_does_not_claim_a_role() -> None: assert "family name" not in amb.detail +@pytest.mark.parametrize("policy", [ + Policy(), + Policy(name_order=FAMILY_FIRST), + Policy(name_order=FAMILY_FIRST_GIVEN_LAST), +]) +def test_chained_particle_detail_is_order_invariant(policy: Policy) -> None: + # _group's emitter is the reason the docs can scope leading-particle + # DESTINATIONS to the default order without qualifying this text: + # it names no field, and the chain it reports is a grouping-stage + # decision taken before any role exists. "Dr. Van Johnson" takes the + # chained branch under every order, so the string is the same one + # three times -- pin it, or the invariant is only an intention. + n = Parser(policy=policy).parse("Dr. Van Johnson") + assert (n.given, n.family) == ("", "Van Johnson") + (amb,) = n.ambiguities + assert amb.kind is AmbiguityKind.PARTICLE_OR_GIVEN + assert amb.detail == ( + "'Van' was chained onto the following name piece; " + "it is also a given name in other names") + + def test_each_suffix_or_name_branch_describes_itself() -> None: # one kind, two causes: the acronym branch turns on periods, the # roman-numeral branch turns on the letter being a numeral. Sharing From 32c545e3e15d24d78d5fa8cc76e0f4bab15e4f65 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 13:27:28 -0700 Subject: [PATCH 4/5] Say what membership decides, not what it chains (#355) The previous sweep replaced a field-destination error with a MECHANISM error at two of its six sites, and missed two more instances of its own defect class. `_types.py`'s `PARTICLE_OR_GIVEN` documents a KIND, and this kind has two emitters: `_assign`'s lone leading particle and `_group`'s prefix chain, when a title shifts the particle off index 0. All three of the rewrite's claims were false for the second. Measured: Dr. Van Johnson default family='Van Johnson' Dr. Van Johnson FAMILY_FIRST family='Van Johnson' Dr. Van Johnson FF_GIVEN_LAST family='Van Johnson' detail (all three, identical): 'Van' was chained onto the following name piece; it is also a given name in other names It WAS chained; it lands in `family` under every order, not just the default; and its detail names no field, so "detail names it rather than the kind" does not describe it. Worse, the rewrite added an explicit denial of chaining ("rather than as a particle chaining onto what follows") to a kind whose other emitter fires only when chaining happened. The docstring now covers both shapes and says what distinguishes them -- one was left standing alone and carries the role assignment gave it, the other was claimed by the chain and names no field because grouping runs before roles exist. `_lexicon.py`'s `particles_ambiguous` said a member "stays a name piece of its own instead of chaining onto what follows", which implies a non-member chains. None does. `_group.py:204` is `if k == 0 or not prefix(k): continue`, so the prefix chain skips index 0 before membership is ever consulted, and both group into two pieces: de Mesnil -> [['de'], ['Mesnil']] (never-given) van Gogh -> [['van'], ['Gogh']] (ambiguous) What produces family='de Mesnil' under the default is `post_rules` rule 1b, a role fold AFTER assignment -- the rule #359 is about. That claim also contradicted `config/particles.py:85-87`, the site the issue named as the pattern to follow, leaving two rendered API docstrings disagreeing; and it is the same claim `c647381` says it measured false and deleted from `customize.rst`. Removed there, planted here. It now names what membership actually decides: the ambiguity report under either order, and the default order's fold for a non-member (with the degenerate bare "de" that rule 1b's middle/family guard leaves alone). `docs/usage.rst`'s "where the pieces land follows that order instead" covered both branches of the preceding sentence, including the never-given one, so it asserted exactly what the PR said it did not. It is wrong in substance too -- "de Mesnil" under FAMILY_FIRST is not those pieces relocated, it is a fold that never fires. It now says only that the destinations shown are the default order's and that `name_order` is what decides them. `docs/concepts.rst`: an inserted em-dash left "which is the right call for the actor Van Johnson" binding to "the start of a surname" rather than to the reading actually taken -- i.e. backwards. The chosen reading and the right-call/wrong-one pair are adjacent again, and the order-dependence moved to its own parenthesis instead of splitting them. Two sites the sweep missed, both its own defect class: _group.py:213 "Van Johnson" -> given nine lines above the emitter it examined AGENTS.md:179 "the particle stayed the GIVEN name" Dr. Van Jr. default given='Van' Dr. Van Jr. FAMILY_FIRST family='Van' #354 fixed the sibling at AGENTS.md:136 and this one was missed. `docs/release_log.rst` said `SUFFIX_OR_NAME` "has always named the part it declined". It names both -- `read as a family name rather than a post-nominal` -- and the new particle detail names only the part it took. The bullet now says which is which. Not touched, and not this change's business: `customize.rst`'s "no given name at all" after a family comma and `usage.rst:746-749`'s "more likely reading" are pre-existing; the 2.2.0 preamble is finalized at release time; and what FAMILY_FIRST should mean for a never-given Latin particle is #359's open question, so nothing here asserts an answer to it. pytest 3117 passed, 20 skipped, 11 xfailed mypy Success: no issues found in 103 source files ruff All checks passed! sphinx -b html build succeeded, 0 warnings sphinx -b doctest 223 tests, 0 failures Co-Authored-By: Claude Opus 5 --- AGENTS.md | 2 +- docs/concepts.rst | 13 +++++++------ docs/release_log.rst | 2 +- docs/usage.rst | 9 +++++---- nameparser/_lexicon.py | 22 ++++++++++++++-------- nameparser/_pipeline/_group.py | 3 ++- nameparser/_types.py | 21 ++++++++++++++------- 7 files changed, 44 insertions(+), 28 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c061a25a..9df9b0b5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -176,7 +176,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. Sanctioned deviation, facade layer only: `HumanName` and the shim `Constants` organize by v1 concern groups (`# -- render defaults --`, `# -- config / parsing --`, `# -- fields --`, ..., dunders and pickle last) — the classes mirror v1's own surface and die in 3.0; the canonical order still binds every core type. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker** — nameparser ships `py.typed`. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. -- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all 39 ambiguous particles on "Dr. Van Jr.", where the particle stayed the GIVEN name and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is deliberately not emitted on the `FAMILY_COMMA` path and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece and in `_group` when a title shifts it off index 0 and the prefix chain claims it, so both report; for two years only the first did. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. +- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel in `_assign`, the delimiter escape's follow-up in `classify` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge(k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all 39 ambiguous particles on "Dr. Van Jr.", where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`) before recording. Structure also structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is deliberately not emitted on the `FAMILY_COMMA` path and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct. **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece and in `_group` when a title shifts it off index 0 and the prefix chain claims it, so both report; for two years only the first did. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission (see the comma paths in concepts.rst) over emitting on input nobody finds ambiguous. - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). diff --git a/docs/concepts.rst b/docs/concepts.rst index 110c02f1..59c8fa78 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -159,12 +159,13 @@ Some calls are irreducibly ambiguous — both readings are legitimate, and no amount of rule-tuning resolves them without breaking some other name. Those surface as entries on ``ParsedName.ambiguities`` instead of being silently guessed away. The canonical example: a leading "Van" -reads as a name of its own — the given name, under the default name -order — rather than as the start of a surname, which is the right call -for the actor Van Johnson and the wrong one for a bare "Van Buren", -and nothing in the two-word shape distinguishes them. So the parse -records a ``particle-or-given`` ambiguity alongside its answer. You -can inspect ``ambiguities`` to decide, case by case, whether your data +reads as a name of its own rather than as the start of a surname — +the right call for the actor Van Johnson, the wrong one for a bare +"Van Buren", and nothing in the two-word shape distinguishes them — so +the parse records a ``particle-or-given`` ambiguity alongside its +answer. (Which name that piece then becomes is a separate question, +and ``name_order``'s: the given name under the default.) You can +inspect ``ambiguities`` to decide, case by case, whether your data needs a second look. An ambiguity records a *decision*, not a word. The same token in a diff --git a/docs/release_log.rst b/docs/release_log.rst index e1946a9c..6ba2086b 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -22,7 +22,7 @@ Release Log **Behavior Changes** - - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, the way ``SUFFIX_OR_NAME`` has always named the part it declined. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355) + - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, reading the role off the assigned token the way ``SUFFIX_OR_NAME`` already did -- that kind names both parts (``read as a family name rather than a post-nominal``), while this one names only the part it took. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355) **Deprecations** diff --git a/docs/usage.rst b/docs/usage.rst index 0cf93389..59007b76 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -117,10 +117,11 @@ names together as easily as two surnames: Position matters in exactly one place: the start of a name. A particle there has no surname to attach to yet, so what decides the reading is -whether it is one that can double as a given name. Under the default -name order it either becomes the given name or turns the whole name -into a surname; under another ``name_order`` — see :doc:`customize` — -where the pieces land follows that order instead: +whether it is one that can double as a given name. Where the pieces +then land is ``name_order``'s question — see :doc:`customize` — and +the destinations below are the default given-first order's: the +particle either becomes the given name or turns the whole name into a +surname: .. doctest:: diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index 2bde8886..c8611df1 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -332,14 +332,20 @@ class Lexicon: #: ("van", "de", "bin", ...). Full default list: #: :data:`~nameparser.config.particles.PARTICLES`. particles: frozenset[str] = frozenset() - #: Subset of particles that can also BE a given name, which is - #: what makes a leading one a fork rather than a plain particle: - #: it stays a name piece of its own instead of chaining onto what - #: follows. Under the default given-first order that piece is the - #: given name ("Van Johnson", but also "Van Buren"); where it - #: lands under another ``name_order`` is that order's question, - #: not this set's. What membership decides under EITHER order is - #: that the fork is recorded as a particle-or-given ambiguity. + #: Subset of particles that can also BE a given name ("Van + #: Johnson", but also "Van Buren"). Membership decides nothing + #: about chaining: the prefix chain skips index 0 unconditionally + #: and never consults this set, so it leaves a leading particle a + #: piece of its own whether listed or not -- "de Mesnil" groups + #: into two pieces exactly as "van Gogh" does. What membership + #: decides is what becomes of that piece afterwards. Under EITHER + #: ``name_order`` a member records a particle-or-given ambiguity + #: and a non-member records none; under the default given-first + #: order a non-member is additionally folded back into the family + #: name once roles exist, so the whole name is the surname ("de + #: Mesnil" -- a bare "de", with nothing to fold into, is left + #: alone). Which field each piece lands in is ``name_order``'s + #: question, not this set's. #: No constant of its own -- the default derives #: as particles minus #: :data:`~nameparser.config.particles.NON_GIVEN_NAME_PARTICLES` diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index fd4cadd9..daad03ef 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -211,7 +211,8 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), j += 1 # The other half of PARTICLE_OR_GIVEN. _assign reports the # fork when an ambiguous particle stays a lone leading piece - # ("Van Johnson" -> given); the chain here takes the + # ("Van Johnson" -> given under the default order, family + # under FAMILY_FIRST); the chain here takes the # opposite branch whenever a title shifts it off index 0 # ("Dr. Van Johnson" -> family "Van Johnson"). A fork whose # two sides are decided in different stages needs an emitter diff --git a/nameparser/_types.py b/nameparser/_types.py index 75bc7512..77e941f4 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -356,13 +356,20 @@ class AmbiguityKind(StrEnum): #: Smith B"). Which name part was declined depends on position and #: ``name_order``, so ``detail`` names it rather than the kind. SUFFIX_OR_NAME = "suffix-or-name" - #: A leading ambiguous particle was read as a name piece of its - #: own rather than as a particle chaining onto what follows -- the - #: right call for "Van Johnson" (the actor's given name), the - #: wrong one for a bare "Van Buren" (the presidential surname); - #: the two-word shape cannot distinguish them. Which name part it - #: was read as depends on ``name_order`` -- the given name under - #: the default -- so ``detail`` names it rather than the kind. + #: An ambiguous particle at the head of a name is either a + #: particle or a name in its own right -- "Van Johnson" is the + #: actor's given name, a bare "Van Buren" the presidential + #: surname, and the two-word shape cannot distinguish them. Two + #: shapes report this kind, decided in different stages, and + #: ``detail`` is what tells them apart. A particle left standing + #: alone chained nothing and was assigned a role, which ``detail`` + #: names ("read as a given name") -- that role is whatever + #: assignment gave it, so it follows ``name_order`` and any + #: ``script_orders`` entry, which is why the kind cannot name it. + #: A particle a title shifted off the front ("Dr. Van Johnson") + #: was instead claimed by the prefix chain, and ``detail`` says + #: that and names no field at all: grouping runs before roles + #: exist, so that text is the same under every order. PARTICLE_OR_GIVEN = "particle-or-given" #: A nickname/maiden delimiter opened without closing (or closed #: without opening); the text was kept as literal name content, so From 14dae56bad0cb4361b80ce3a2b1116496ad7f1b9 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 9 Aug 2026 13:29:34 -0700 Subject: [PATCH 5/5] Give the leading-particle exception its real reason (#355) The PARTICLES docstring said a leading particle "chains nothing, since it may be a given name instead". The behavior is right and the reason is not: the chain loop skips index 0 unconditionally (_group.py, `if k == 0 or not prefix(k)`), so a never-given particle chains nothing either. Measured -- "de Mesnil" and "van Gogh" both group into two separate pieces, though only one of them could be a given name. The `since` clause offered a contingent rationale for unconditional behavior, which is the shape that gets read as a rule and then relied on. This is the docstring #355 named as the pattern for its sweep, and the sweep's own rewrites now say membership decides nothing about chaining. Leaving the model site giving a membership-flavored reason for it would have left the two disagreeing about why, one commit after they were made to agree about what. Narrower than it first looks, and the narrowing matters: the claim is about the PREFIX chain, not about grouping as a whole. The bound given-name rule does chain a leading piece, and three entries are in both vocabularies -- `parse("Abu Bakr Ahmed")` groups as [Abu Bakr] [Ahmed], a leading ambiguous particle that was chained. It keys on `vocab:bound-given` and never on this set, so the statement holds as written. Co-Authored-By: Claude Opus 5 --- nameparser/config/particles.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/nameparser/config/particles.py b/nameparser/config/particles.py index 0f55a575..7d751185 100644 --- a/nameparser/config/particles.py +++ b/nameparser/config/particles.py @@ -83,8 +83,10 @@ #: joins each following piece until the suffix "MD", giving the family #: name "von bergen wessels", while the same chaining in "Smith, Juan #: de la Cruz" gives the middle name "de la Cruz". A leading -#: particle is the exception and chains nothing, since it may be a given -#: name instead. Where the pieces then land is again a later question, +#: particle is the exception and chains nothing: the chain skips the +#: first piece unconditionally, membership in this set or any other +#: never entering into it. Where the pieces then land is again a later +#: question, #: and this one is ``name_order``'s: under the default given-first order #: a leading :py:data:`NON_GIVEN_NAME_PARTICLES` member makes the whole #: name a family name ("de la Vega"), while a leading particle outside