diff --git a/AGENTS.md b/AGENTS.md index 0124076..2b1a98d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,7 +75,7 @@ logging.getLogger('HumanName').setLevel(logging.DEBUG) The library has two layers: `nameparser/config/` (data) and `nameparser/parser.py` (logic). -**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Five defaults fall out of it, plus a sixth that applies the same not-a-guess standard to specific WORDS rather than to a script (#308's honorific peel, below). Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. And the 间隔号 U+00B7 (#298) is the transcription marker for scripts that HAVE no transcription script: a name it divides (威廉·莎士比亚 — flanked by classified characters on both sides, so Catalan's Gal·la is untouched) keeps its source order and never segments — the orthography names the convention, exactly as pure katakana does, with the divider carrying the signal instead of the script. And a listed CJK honorific glued to the END of a name token is split off it (#308) — 田中さん is 田中 plus さん — on the same orthography-settles-it test, narrowed for the glued position: an entry peels only where it can never end a name, so 씨/님/さん/様/先生 peel while 양/군/氏/博士/殿 stay spaced-only (김지양 and 田中博士 are names, and ~90 Japanese surnames end in 殿) and 君 is in NEITHER set (王君 is a complete Chinese name), though its kana spelling くん peels. Like the nakaguro's tokenize-level separation described next, it is reached by neither policy opt-out — but for its own reason: the vocabulary carries the license itself rather than borrowing the script's, so `segment_scripts` has nothing to say about it. Since #312 it also crosses the FAMILY comma and the 间隔号, which is the more surprising fact, since both of those still stop the surname split standing right beside it: each answers where a name DIVIDES into surname and given, and the peel never asks that question — so `김, 민준씨` reads exactly as the spaced `김 민준씨` does (family 김, given 민준, suffix 씨) while the split stands down as before. Its site is accordingly the name-bearing segment runs — `segments[:2]` under a family comma, and `segments[0]` as before otherwise, the family comma being the one structure that splits the name itself across two runs, with the honorific as often glued to the given side as to the family. That is the whole reach and nothing past it (`김, 민준 지훈씨` peels; `김, 민준, 지훈씨` and `김,, 민준씨` do not, both landing in a third run), and it presumes `segments[1]` is name text, which `segment` does not guarantee — a one-word part before the comma reads as FAMILY_COMMA even when the part after it is entirely suffix-shaped, so `田中さん, V.` puts the scan on `V.` and さん stays in the family name where `田中さん, PhD` gives it up (parity with 1.4.0, a strict/lenient gap that predates #312, and the reason to reach for a shared `suffixy` predicate before widening this). `田中さん, 太郎` is unchanged, and not because of its comma — the honorific there is not at the end of the name, 太郎 is. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. +**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Five defaults fall out of it, plus a sixth that applies the same not-a-guess standard to specific WORDS rather than to a script (#308's honorific peel, below). Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. And the 间隔号 U+00B7 (#298) is the transcription marker for scripts that HAVE no transcription script: a name it divides (威廉·莎士比亚 — flanked by classified characters on both sides, so Catalan's Gal·la is untouched) keeps its source order and never segments — the orthography names the convention, exactly as pure katakana does, with the divider carrying the signal instead of the script. And a listed CJK honorific glued to the END of a name token is split off it (#308) — 田中さん is 田中 plus さん — on the same orthography-settles-it test, narrowed for the glued position: an entry peels only where it can never end a name, so 씨/님/さん/様/先生 peel while 양/군/氏/博士/殿 stay spaced-only (김지양 and 田中博士 are names, and ~90 Japanese surnames end in 殿) and 君 is in NEITHER set (王君 is a complete Chinese name), though its kana spelling くん peels. Like the nakaguro's tokenize-level separation described next, it is reached by neither policy opt-out — but for its own reason: the vocabulary carries the license itself rather than borrowing the script's, so `segment_scripts` has nothing to say about it. Since #312 it also crosses the FAMILY comma and the 间隔号, which is the more surprising fact, since both of those still stop the surname split standing right beside it: each answers where a name DIVIDES into surname and given, and the peel never asks that question — so `김, 민준씨` reads exactly as the spaced `김 민준씨` does (family 김, given 민준, suffix 씨) while the split stands down as before. Its site is accordingly the name-bearing segment runs — `segments[:2]` under a family comma, and `segments[0]` as before otherwise, the family comma being the one structure that splits the name itself across two runs, with the honorific as often glued to the given side as to the family. That is the whole reach and nothing past it (`김, 민준 지훈씨` peels; `김, 민준, 지훈씨` and `김,, 민준씨` do not, both landing in a third run), and whether `segments[1]` is name text at all is now ASKED rather than inferred from the structure — `segment` does not guarantee it, since a one-word part before the comma reads as FAMILY_COMMA even when the part after it is entirely suffix-shaped, and the peel walking into such a run took `V.` for its site, found no listed tail and abandoned (#319). The question is `segment`'s own suffix-comma predicate, lifted into `_vocab.is_wholly_suffix` so the two stages cannot drift: a wholly suffix-shaped second run is declined and the scan stays in `segments[0]`, so `田中さん, V.` and `田中さん, Ph. D.` give さん up as `田中さん, PhD` always did. The test is necessary but NOT sufficient, and the second condition is not decoration: every honorific tail is also a suffix word, so a glued honorific is itself part of what makes its run read as suffix-shaped, and declining a run that holds the ONLY site loses the peel outright. `segments[0]` must therefore offer a peel site of its own before the second run is declined — `이, J.씨` and `선생님, J.씨` pass the suffix test and are scanned anyway, keeping the pre-#319 reading, while `김민준씨, J.씨` has a site on both sides and peels the person's own 씨 rather than the junk one behind the comma. Uniform in the PEEL, that is — where the credential itself lands is `assign`'s question and still differs by spelling (`PhD` → `title`, `V.` → `given`, `Ph. D.` → `suffix`). Not `_is_post_nominal` pluralized: the run predicate says yes both to what the token predicate vetoes (`V.`, `V`, `I` — the class the defect was reported as) and to what the token predicate never sees at all, since `period_joined_vocab` and the delimiter routes are the run predicate's alone (`Msc.Ed.` and `J.씨` reach it that way, and `田中さん, Msc.Ed.` moves with the rest). `Policy(lenient_comma_suffixes=False)` drops this call to the strict token test too — so those three read as name text again and keep the pre-#319 answer, while `Ph. D.` peels under the knob regardless, its merged `phd` passing the strict test. `田中さん, 太郎` is unchanged, and not because of its comma — the honorific there is not at the end of the name, 太郎 is. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. ### Configuration layer (`nameparser/config/`) diff --git a/docs/customize.rst b/docs/customize.rst index 05f4dd1..3e9725b 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -261,7 +261,11 @@ listed below. ``"John Smith, V"`` is John Smith the fifth when ``True`` (default); ``False`` reads ``V`` as a given-name initial instead. Multi-letter suffixes (``III``, ``MD``) are - unaffected. + unaffected. The same test is also one of the two the + glued-honorific peel asks before crossing a family comma + (#319), so the setting reaches CJK names too: ``"田中さん, + V."`` gives family ``田中``, suffix ``さん`` when ``True``, and + family ``田中さん``, given ``V.`` when ``False``. * - ``strip_emoji`` - ``bool`` - Excludes emoji from tokenization — they appear in no field or diff --git a/docs/release_log.rst b/docs/release_log.rst index 848b599..0f1ae43 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -28,8 +28,9 @@ Release Log - Fix 间隔号-divided transcriptions parsing as one unsplit token: U+00B7 — the interpunct Chinese text divides a transcribed foreign name with, ``威廉·莎士比亚`` for William Shakespeare — is now a token separator between characters of a classified script, and a name it divides keeps its source order and is never segmented: the dot is the transcription marker, playing the role pure katakana plays in the kana license. It divides ONLY between classified-script characters, so the Catalan punt volat interior to names like ``Gal·la`` is untouched. The Japanese nakaguro is deliberately not a transcription marker — ``高橋・一郎`` is roster formatting, 姓・名, and keeps its family-first reading — so a transcription typed with the wrong dot (``威廉・莎士比亚``) reads by the convention of the codepoint it was typed with; like the spaced form, only the Chinese dot rescues it. Rendering stays space-joined (``str(HumanName("威廉·莎士比亚"))`` is ``"威廉 莎士比亚"``); a custom ``string_format`` such as ``"{first}·{last}"`` reinstates the dot (#298) - Fix spaced CJK postnominal honorifics parsing as name parts: 씨, 박사, 선생님, 교수님, 군, 양 (Korean — standardly written as their own token), 先生, 女士, 小姐, 博士, 教授 (Chinese, with 先生/博士/教授 shared with Japanese), and 様, 氏 (Japanese) now route to ``suffix``, so ``王小明 先生`` reads family ``王小明`` where the family-first default had confidently made 先生 the given name. Whole-token matching, which also reaches a glued surname+honorific token, since segmentation splits off the surname first (``김씨`` reads family 김, suffix 씨). The glued forms whole-token matching cannot reach are handled by the peel described below (closes #307) - Fix glued CJK honorifics parsing as part of the name: an honorific written against the name — ``田中さん``, ``山田太郎様``, ``김민준씨``, ``김민준님``, ``王小明先生`` — is now split off the end of the last name token and routed to ``suffix``, where it had been swallowed by the name (the whole of ``田中さん`` was the family name; ``김민준씨`` gave given 민준씨). The peeled name then goes through the ordinary machinery, so the Korean split still happens (``김민준씨`` → family 김, given 민준, suffix 씨) and the ``田中さん`` case stops misreading as a kana-licensed composite. Only entries that can never END a name peel — 씨, 님, 박사, 박사님, 선생님, 교수님, さん, さま, くん, ちゃん, 様, 先生, 教授, 女士, 小姐 — while 양, 군, 氏, 博士 and 殿 are recognized in their spaced form only, because 김지양 is a given name, 田中博士 is Tanaka Hiroshi as readily as Doctor Tanaka, and some ninety Japanese surnames end in 殿 (鵜殿, 真殿); 君 is recognized in neither form, since 王君 is a complete Chinese name, though its kana spelling くん peels. Seven entries are new vocabulary in this change (さん, さま, くん, ちゃん, 殿, 님, 박사님), so their spaced forms route to ``suffix`` too (``田中 さん``, ``田中 殿``, ``김민준 님``, ``김민준 박사님``). 박사님 closes a gap in the shipped set rather than opening new ground: 선생님 and 교수님 shipped in 2.1 without it, so ``김민준박사님`` stranded 박사 in the given name and the spaced ``김민준 박사님`` came back as two suffixes for one honorific. Exactly one honorific peels off a token, and every entry is a whole honorific rather than a part of one. A token that is nothing but an honorific is no longer taken apart either — ``선생님`` and ``박사`` now stay whole where hangul segmentation had split them 선 + 생님 and 박 + 사, since 선 and 박 are listed surnames — which is **default-on** in its own right, hangul segmentation being a default. A configured segmenter benefits twice over: it is handed the name without the honorific, and the honorific the peel just cut off does not then look to it like a boundary the writer drew — so ``parser_for(locales.JA, segmenter=ja_segmenter())`` reads ``山田太郎様`` as family 山田, given 太郎, suffix 様. Worth knowing before you upgrade: that exemption is what makes a GLUED honorific stop protecting a name from division, so a family name written alone with one — ``田中さん`` — now divides the way bare ``田中`` already did (family 田, given 中, suffix さん). It is exactly and only the peeled tail that is exempt. A SPACED honorific is a token boundary its writer typed, and anything standing beside a name calls the segmenter off, so the name is left as written: ``田中 さん`` and ``佐藤 氏`` keep family 田中 and 佐藤 under the pack — the division the pack gives them without this change, whichever field the honorific itself lands in. That is the conservative reading rather than a claim about intent: a spaced honorific cannot be told apart from a spaced given name by position, and counting it as one keeps four real surnames whole (``佐藤 氏``, ``田中 様``, ``鈴木 先生``, ``中村 教授``) at the price of the one division it then declines to make (``山田太郎 様``). Writing the honorific spaced is therefore an opt-out in its own right on the SEGMENTER path, alongside declining the pack or the segmenter. It is no lever where the VOCABULARY divides the name, the two spellings agreeing exactly there — ``김민준 씨`` and ``김민준씨`` both give family 김, given 민준, suffix 씨, as do ``王小明 先生`` and ``王小明先生`` under the Chinese pack — and Korean data has no pack to decline either, hangul segmentation being on by default. **Default-on: changes parse output for glued CJK honorific forms**, through ``HumanName`` as well as the 2.0 API (closes #308) - - Fix a comma or a 间隔号 stopping the glued-honorific peel: an honorific written against the name is now split off and routed to ``suffix``, so ``김, 민준씨`` gives family 김, given 민준, suffix 씨 — the same as the spaced ``김 민준씨`` — and likewise ``田中, 太郎さん``, which also matches its spaced form. ``田中さん, PhD`` and ``威廉·莎士比亚さん`` peel too; the first of those does *not* otherwise match its spaced form, since ``田中さん PhD`` leaves PhD in ``suffix`` beside さん while after a comma it reads as a ``title`` — where the credential lands is the comma's business, not the peel's. Previously each of these left the honorific inside the name. A comma no longer switches the peel off; what it does now is say which runs of the name to look in, and those are the two around a family comma, an honorific being as often glued to the given name as to the family. Anything past those two runs is out of reach, which is the one limit worth knowing: ``김, 민준 지훈씨`` peels, while ``김, 민준, 지훈씨`` (a third run) and ``김,, 민준씨`` (a doubled comma, which puts the name in a later run) do not. The reach also rests on the second run being name text, and a one-word part before the comma reads as a family comma even when the part after it is entirely suffix-shaped — so ``田中さん, V.`` keeps さん in the family name where ``田中さん, PhD`` gives it up. Same credential, opposite answer, and unchanged from 1.4.0 in that spelling. The 间隔号 does not switch the peel off either. Both marks say where a name divides into surname and given, and an honorific is not part of the name in either reading. The surname split still stands down for both, unchanged — a comma still means the writer said where the family name ends, and the 间隔号 still marks a transcription. ``田中さん, 太郎`` is unaffected, because the honorific there is not at the end of the name: 太郎 is. **Default-on**, and it reaches ``HumanName`` too (closes #312) - - Fix an ASCII period after a CJK honorific stopping it being recognized: ``씨.``, ``様.``, ``氏.``, ``님.``, ``군.``, ``양.`` and ``殿.`` now route to ``suffix`` like their periodless spellings, where the trailing period had left them inside the name — the family name in ``"김민준 씨."``, the given name in ``"김민준, 씨."``. The cause was v1's initial regex, ``^(\w\.|[A-Z])?$`` (``REGEXES["initial"]``, still public v1 API), whose ``\w`` is Unicode-aware and so matched a hangul syllable or a Han ideograph as readily as a letter; the strict suffix test applies that as a veto (``V.`` in ``"John V. Smith"`` is a middle initial, not roman five), and a veto written for Latin was being asked of scripts it was never about. The cost ran past the honorific itself: because the vetoed token read as name text, the glued-honorific peel's scan back for its site stopped at it instead of stepping over it, took it as the site, found no honorific at the end of it and gave up — so ``"田中さん, 様."`` kept ``さん`` inside the family name while ``"田中さん, 様"`` peeled it. Measured against 1.4.0, ``"김민준, 씨."`` and ``"田中さん, 様."`` were returning exactly what 1.x returns, so the honorific work earlier in this release had a hole in it wherever the honorific was written with a period. An initial is a single LETTER standing in for a name, and Han ideographs, hangul syllables and kana are morphemes and syllables rather than letters, so the veto now asks its question only of the scripts where it means something. Alphabets keep their initials untouched — ``"А. С. Пушкин"``, ``"م. الفارسي"`` and ``"Ա. Խաչատրյան"`` are unaffected, and so is the Ukrainian conjunction entry below, where a punctuated ``Й.`` still outranks the conjunction ``й``. The public ``initial`` tag follows the same line: ``씨.`` no longer carries it. Read ``period`` strictly here: the fix is scoped to the ASCII full stop U+002E, because that is the only period ``_normalize`` strips. The fullwidth U+FF0E and the ideographic U+3002 (with its halfwidth twin U+FF61) — the stops a CJK writer is likelier to type — leave the honorific unmatchable by the vocabulary lookup, which runs before the veto is ever consulted, so ``"김민준 씨．"`` still reads the honorific as the family name. That is a separate, still-open gap in ``_normalize`` rather than in the veto: those spellings parse identically before and after this change, and widening the strip is follow-up work. **Default-on**, and it reaches ``HumanName`` too (#320) + - Fix a comma or a 间隔号 stopping the glued-honorific peel: an honorific written against the name is now split off and routed to ``suffix``, so ``김, 민준씨`` gives family 김, given 민준, suffix 씨 — the same as the spaced ``김 민준씨`` — and likewise ``田中, 太郎さん``, which also matches its spaced form. ``田中さん, PhD`` and ``威廉·莎士比亚さん`` peel too; the first of those does *not* otherwise match its spaced form, since ``田中さん PhD`` leaves PhD in ``suffix`` beside さん while after a comma it reads as a ``title`` — where the credential lands is the comma's business, not the peel's. Previously each of these left the honorific inside the name. A comma no longer switches the peel off; what it does now is say which runs of the name to look in, and those are the two around a family comma, an honorific being as often glued to the given name as to the family. Anything past those two runs is out of reach, which is the one limit worth knowing: ``김, 민준 지훈씨`` peels, while ``김, 민준, 지훈씨`` (a third run) and ``김,, 민준씨`` (a doubled comma, which puts the name in a later run) do not. The second of those two runs counts only where it is name text, which is a question the parser asks rather than reads off the comma — see the next entry. The 间隔号 does not switch the peel off either. Both marks say where a name divides into surname and given, and an honorific is not part of the name in either reading. The surname split still stands down for both, unchanged — a comma still means the writer said where the family name ends, and the 间隔号 still marks a transcription. ``田中さん, 太郎`` is unaffected, because the honorific there is not at the end of the name: 太郎 is. **Default-on**, and it reaches ``HumanName`` too (closes #312) + - Fix a glued honorific staying inside the name when the whole post-comma remainder is a credential: ``田中さん, V.`` and ``田中さん, Ph. D.`` now give up さん to ``suffix`` the way ``田中さん, PhD`` already did, and so do ``김민준씨, V.``, ``王小明先生, Ph. D.`` and ``山田太郎様, Jr. V.``. The peel looks in the two runs around a family comma (the entry above), and it had been taking the second of those for name text on the strength of the comma alone. A comma does not say that much: the suffix-comma reading also wants more than one word ahead of the comma, so a one-word name followed by nothing but credentials reads as a family comma anyway — which is exactly the shape ``田中さん, V.`` has. The scan then walked into the credentials, took ``V.`` for the honorific's site, found no honorific on the end of it and abandoned the peel without a word. Whether a post-comma run is credentials is now asked with the same test that decides the comma structure itself, and where the answer is yes the run is declined and the scan stays with the name. Declining takes a second condition, because that test is not independent of the peel: every honorific tail is also a suffix word, so a glued honorific is itself part of the evidence that the run it sits in is suffix-shaped. The run is therefore declined only where the part *before* the comma offers a peel site of its own — otherwise declining would throw away the only site there is, and ``이, J.씨`` would lose its given name, where it in fact still reads given ``J.``, family ``이``, suffix ``씨``. Where both parts offer a site the decline stands and the person's own honorific is the one peeled: ``김민준씨, J.씨`` now gives family 김민준 with ``씨, J.씨`` in ``suffix``, where before it peeled the 씨 off the junk ``J.씨`` behind the comma and left 김민준씨 whole. One credential in three spellings, one answer — from the *peel*: where the credential itself lands is the comma's business as before and still differs by spelling, ``PhD`` reading as a ``title``, ``V.`` as a ``given``, and ``Ph. D.`` joining さん in ``suffix``. ``Policy(lenient_comma_suffixes=False)`` keeps the old reading for the initial-shaped credentials the knob is about — with it set, ``V.``, ``V`` and ``I`` are a given name again and the family name is the whole ``田中さん`` — but it is no blanket freeze of the old behavior: ``田中さん, Ph. D.`` peels under the knob too, the credential pair merging to a form the strict test accepts either way. Nothing Latin-script moves, since nothing there had an honorific to peel, and no name that peeled before stops peeling — that second one is what the site condition above buys, and it is measured rather than reasoned. What did change wherever the run *is* declined, answer or no answer, is which runs the peel reads: ``田中さん, PhD`` and ``田中さん, 様.`` reached さん before by stepping over the post-comma token the scan had walked onto, and reach it now by never looking at that run — ``様.`` being a listed honorific rather than a credential, and the second one this name has no use for. **Default-on**, and it reaches ``HumanName`` too (closes #319) + - Fix an ASCII period after a CJK honorific stopping it being recognized: ``씨.``, ``様.``, ``氏.``, ``님.``, ``군.``, ``양.`` and ``殿.`` now route to ``suffix`` like their periodless spellings, where the trailing period had left them inside the name — the family name in ``"김민준 씨."``, the given name in ``"김민준, 씨."``. The cause was v1's initial regex, ``^(\w\.|[A-Z])?$`` (``REGEXES["initial"]``, still public v1 API), whose ``\w`` is Unicode-aware and so matched a hangul syllable or a Han ideograph as readily as a letter; the strict suffix test applies that as a veto (``V.`` in ``"John V. Smith"`` is a middle initial, not roman five), and a veto written for Latin was being asked of scripts it was never about. The cost ran past the honorific itself: because the vetoed token read as name text, the glued-honorific peel's scan back for its site stopped at it instead of stepping over it, took it as the site, found no honorific at the end of it and gave up — so ``"田中さん 様."`` kept ``さん`` inside the name, reading given ``田中さん`` and family ``様.``, while ``"田中さん 様"`` peeled it. The comma form ``"田中さん, 様."`` reached the same scan when this was written; since the #319 entry above it no longer does — its post-comma run is now declined as suffix-shaped before the scan begins, so ``様.`` gets to ``suffix`` through classification rather than by being stepped over. Measured against 1.4.0, ``"김민준, 씨."`` and ``"田中さん, 様."`` were returning exactly what 1.x returns, so the honorific work earlier in this release had a hole in it wherever the honorific was written with a period. An initial is a single LETTER standing in for a name, and Han ideographs, hangul syllables and kana are morphemes and syllables rather than letters, so the veto now asks its question only of the scripts where it means something. Alphabets keep their initials untouched — ``"А. С. Пушкин"``, ``"م. الفارسي"`` and ``"Ա. Խաչատրյան"`` are unaffected, and so is the Ukrainian conjunction entry below, where a punctuated ``Й.`` still outranks the conjunction ``й``. The public ``initial`` tag follows the same line: ``씨.`` no longer carries it. Read ``period`` strictly here: the fix is scoped to the ASCII full stop U+002E, because that is the only period ``_normalize`` strips. The fullwidth U+FF0E and the ideographic U+3002 (with its halfwidth twin U+FF61) — the stops a CJK writer is likelier to type — leave the honorific unmatchable by the vocabulary lookup, which runs before the veto is ever consulted, so ``"김민준 씨．"`` still reads the honorific as the family name. That is a separate, still-open gap in ``_normalize`` rather than in the veto: those spellings parse identically before and after this change, and widening the strip is follow-up work. **Default-on**, and it reaches ``HumanName`` too (#320) - Fix NFD-decomposed input missing the East Asian defaults entirely: script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS — where decomposed text is routine — gets the same order rule as its composed twin, which it silently did not before. Segmentation MATCHING deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha worth stating plainly: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name — compare NFC-normalized text when comparing across encodings (#272) - Fix the Ukrainian conjunction ``й`` not joining the pieces around it: it is the euphonic alternate of ``і``, the two chosen by the surrounding vowel and consonant rather than by meaning (``"Олесь і Олена"`` but ``"Марія й Петро"``), so real Ukrainian data carries both spellings and shipping only ``і`` recognized just one of them. ``"Олесь й Олена Коваленки"`` now gives given ``"Олесь й Олена"`` where the ``й`` previously landed in ``middle``. Same treatment as the ``и``/``і`` entries added in 2.0.0, single-letter carve-out included: the conjunction joins only once the name has enough pieces, and a punctuated initial still wins, so ``"Й. Сліпий"`` is unaffected. Raised in a comment on #267 diff --git a/docs/usage.rst b/docs/usage.rst index 7e3b3d4..02b6a1e 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -312,10 +312,21 @@ same as ``김 민준씨``. What a comma does instead is say which runs are the name: the two around a family comma, an honorific being as often glued to the given name as to the family. Nothing past those two is in reach — ``김, 민준 지훈씨`` peels, ``김, 민준, 지훈씨`` does not — and -the reach rests on the second run being name text, so where it is not -the peel is abandoned: in ``田中さん, V.`` the scan lands on ``V.``, -which is neither a name nor an honorific, and さん stays in the family -name where ``田中さん, PhD`` gives it up. What those marks do stop is +the second of those runs counts only where it is name text. A run that +is nothing but credentials is not, whatever the comma looked like, and +the same test that decides the comma structure decides this too: the +run is left alone and the honorific is found in the name. Being +credentials is necessary but not enough, because the test cannot help +counting the honorific itself — a glued honorific is a suffix word, +so it is part of what makes its own run read as credentials. The run +is passed over only when the part *before* the comma has an honorific +of its own to give up; otherwise passing it over would discard the +only one there is. So ``田中さん, V.`` and ``田中さん, Ph. D.`` +give さん up exactly as +``田中さん, PhD`` does — though where the credential itself lands is +the comma's business rather than the peel's, and still differs by +spelling (``title`` for ``PhD``, ``given`` for ``V.``, and ``Ph. D.`` +in ``suffix`` beside さん). What those marks do stop is the *split*, which is a different question and still theirs to answer. ``田中さん, 太郎`` is unchanged, and not because of its comma: the honorific there is not at the end of the name, 太郎 is. diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index dcd51fa..6a29740 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -31,7 +31,11 @@ from nameparser._pipeline._vocab import delimiter_cores from nameparser._types import AmbiguityKind, Role -# the credential-pair regexes live in _vocab (shared with segment) +# the credential-pair regexes live in _vocab, whose own +# is_wholly_suffix merges the same pair -- and since #319 that +# predicate has TWO callers to stay in sync with, segment's +# suffix-comma structure test and script_segment's decline of a +# wholly-suffix post-comma run, both of which see the merged reading Piece = list[int] diff --git a/nameparser/_pipeline/_script_segment.py b/nameparser/_pipeline/_script_segment.py index f9ae7a0..a3ddde4 100644 --- a/nameparser/_pipeline/_script_segment.py +++ b/nameparser/_pipeline/_script_segment.py @@ -12,8 +12,28 @@ scored under the confidence floor). Reads: Policy.segment_scripts, Lexicon.surnames, Lexicon.honorific_tails, ParseState.segmenter, and Lexicon suffix -vocabulary (via _vocab.is_suffix_strict) -- both the peel's scan-back -and the surname site ask whether a token is a post-nominal. +vocabulary through TWO predicates, which are NOT one another's +singular and plural. _vocab.is_suffix_strict asks whether a single +token is a post-nominal, initial veto included (the peel's scan-back +and the surname site). _vocab.is_wholly_suffix asks segment's own +suffix-comma question of a whole RUN, through the POLICY-selected +token test plus period_joined_vocab, delimiter handling and the +Ph./D. merge -- so the run predicate says yes both to tokens the +token predicate VETOES ("V.", "V", "I") and to tokens it never sees +as suffixes at all ("Msc.Ed.", "J.씨", which reach it through +period_joined_vocab). The initial-shaped words are the class #319 was +ABOUT, not the whole of the disagreement, and the extra routes listed +above are where the rest of it comes from; +test_is_wholly_suffix_is_not_the_plural_of_is_post_nominal pins the +"V." half. The run predicate is how the peel declines a run that is +not name text (#319), but only where the name's own run offers a +site, since a glued honorific is itself part of what makes a run read +as suffix-shaped; it owns two further Policy fields +(lenient_comma_suffixes picks the strict or lenient token test -- it +flips "田中さん, V." -- and extra_suffix_delimiters both counts a bare +delimiter-core token as a suffix and splits a token on a core, the +split being the half that flips "田中さん, Jr./V." under {"/"} and the +bare core the half that flips "田中さん, /"). Unspaced CJK names give tokenize no separator to find, so this stage inserts the missing token boundary by vocabulary: the first token @@ -134,11 +154,14 @@ import dataclasses import functools +from collections.abc import Sequence from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, ) -from nameparser._pipeline._vocab import effective_script, is_suffix_strict +from nameparser._pipeline._vocab import ( + effective_script, is_suffix_strict, is_wholly_suffix, +) from nameparser._types import AmbiguityKind, Segmentation, Span #: Marks the tail token the peel below MANUFACTURED, so the segmenter's @@ -307,6 +330,49 @@ def _is_post_nominal(state: ParseState, i: int) -> bool: return is_suffix_strict(state.tokens[i].text, state.lexicon) +def _peel_site(state: ParseState, flat: Sequence[int], + tails: frozenset[str]) -> tuple[int, int] | None: + """Where a peel would land in the token run `flat`: the index of the + token to cut and the LENGTH of the listed tail to take off it, or + None where that run offers no peel. + + Two callers, one answer, which is the point of naming it. The peel + itself asks it once, of the runs it decided to scan. The gate above + that decision asks it of segments[0] ALONE, to find out whether + declining the second run would cost the only site. Sharing the scan + with the peel rather than approximating it there is what makes the + gate's answer mean what it says: a gate that merely looked for a + token ENDING in a listed tail would count a token that is a tail + entire, which the scan-back skips as a site -- "선생님, J.씨" would + decline on a site the peel then cannot use, and lose the peel in + exactly the way the gate exists to prevent. (Only the lone-token + shape of that divergence is reachable from the gate, and the gate's + own other conjunct is what makes that so: SUFFIX_COMMA wants BOTH + a suffix-shaped second run and more than one word before the + comma, and the gate is asked only where the first of those is + already true -- so a run this call ever sees under FAMILY_COMMA + failed on the second, and segments[0] holds at most one token. The + word-count alone would not say that: "Dr 김민준, 지훈" has two + words before the comma and is FAMILY_COMMA.) + + Callers must pass a NON-EMPTY `tails` -- _longest_entry's + precondition, which the stage's own early return supplies.""" + i = next((j for j in reversed(flat) + if not _is_post_nominal(state, j)), None) + if i is None: + # nothing but post-nominals, or no tokens at all + return None + text = state.tokens[i].text + # range/cap construction identical to the surname match below, and + # for the same two reasons: longest-first, and a len-1 cap that + # makes the offset interior by construction (_split's contract). + cap = min(_longest_entry(tails), len(text) - 1) + for length in range(cap, 0, -1): + if text[-length:] in tails: + return i, length + return None + + def _peel_honorific_tail(state: ParseState) -> ParseState: """#308: split a listed honorific off the END of the name's last NON-POST-NOMINAL token -- 田中さん -> 田中 + さん -- and let @@ -397,49 +463,93 @@ def _peel_honorific_tail(state: ParseState) -> ParseState: # ("김민준씨 Jr." is one run of two tokens), which is why the # scan-back above steps over such a token rather than simply never # reaching it. - # Reaching past the name's own runs is a live bug, not tidiness: - # segment admits a post-comma run on is_suffix_lenient while - # _is_post_nominal asks is_suffix_strict, and the initial-shaped - # suffix words ("V.", "V", "I") fall in that gap -- as a peel site - # such a token ends in no tail and silently abandons the peel, so - # "Dr 김민준씨, V." would strand 씨 in the given name. A junk tail - # is worse: "Dr 김민준씨, Jr., 박씨" would peel the 씨 off 박씨 and - # leave the person's own glued. Not only a counterfactual, either: - # Policy(lenient_comma_suffixes=False) drops segment's post-comma - # test to the strict one, which reads that same input as - # FAMILY_COMMA -- so the scan reaches "V." through the sanctioned - # two-run crossing and strands 씨 for real, in family - # "Dr 김민준씨". Same gap, reached by a documented knob rather - # than by widening this line; see the case row - # ja_honorific_glued_family_comma_suffixy_second_run, which is the - # DEFAULT-policy shape of it. + # The second run is only NAME text when segment read it as one, + # which the structure alone does not say: SUFFIX_COMMA also wants + # more than one word before the comma, so a one-word part turns a + # wholly suffix-shaped remainder into FAMILY_COMMA anyway ("田中さん, + # V." is that input). So ask segment's own predicate instead of + # inferring the answer from the structure it produced (#319). + # is_wholly_suffix, NOT the plural of _is_post_nominal: the two + # disagree on the initial-shaped suffix words ("V.", "V", "I"), + # which is the class #319 was reported about, and on everything + # the run predicate's extra routes reach and the token predicate + # does not ("Msc.Ed." and "J.씨" by period_joined_vocab, both of + # which this change also moves). Reaching into such a + # run put the site on "V.", which ends in no listed tail, so the + # peel silently abandoned and さん stayed glued to the family -- + # while "田中さん, PhD" peeled all along, because "PhD" satisfies + # the strict test and the scan-back stepped over it. One credential, + # two spellings, two answers FROM THE PEEL -- and the peel's answer + # is the only one that moved: the peeled remainder is 田中 and lands + # in family under every spelling, while where the CREDENTIAL lands + # is assign's question and still differs ("PhD" a title, "V." a + # given, "Ph. D." a suffix beside さん). + # A junk tail is the worse shape of the same reach: in + # "김민준씨, J.씨" the site lands on the junk "J.씨", so master + # peeled THAT 씨 and left the person's own glued inside family + # "김민준씨". Reachable only where the run is genuinely + # suffix-shaped, which "J.씨" is by period_joined_vocab; a run of + # ordinary name text is scanned on purpose, and a junk tail further + # out than the second run is held off by the scope rule instead + # ("Dr 김민준씨, Jr., 박씨" is SUFFIX_COMMA with 박씨 in a third + # run, so it never reaches here at all). + # An EMPTY second run stays in scope and contributes nothing: + # is_wholly_suffix is False on it by its own contract (v1 read + # "Doe,, Jr." as a family comma), which is the reading this line + # wants anyway -- flattening an empty run adds no site. + # Policy(lenient_comma_suffixes=False) keeps the old answer for the + # INITIAL-shaped suffixes specifically, which is where the + # strict/lenient gap lives: the knob drops this call to the strict + # predicate too, so is_wholly_suffix(["V."]) is False, the run reads + # as name text, it IS scanned, and the peel is abandoned on "V." as + # before -- family "田中さん", given "V.". It is not a blanket + # freeze of the old behavior, and "田中さん, Ph. D." is the input + # that shows the difference: the Ph./D. merge folds that pair to a + # form is_suffix_strict accepts, so the run is declined and the peel + # fires under the strict knob as well. # Flattening the SEGMENTS rather than state.tokens is load-bearing # too: extracted nickname and maiden content is in tokens but in NO # segment, and scanning tokens would put the peel site on a # nickname ("김민준씨 (Jimmy)" -> the site becomes Jimmy and # nothing peels). ko_honorific_glued_given_nickname pins that. - runs = (state.segments[:2] if state.structure is Structure.FAMILY_COMMA - else state.segments[:1]) - flat = [j for seg in runs for j in seg] - i = next((j for j in reversed(flat) - if not _is_post_nominal(state, j)), None) - if i is None: + # And declining takes a SECOND condition: segments[0] must hold a + # peel site of its own. is_wholly_suffix reaches period_joined_vocab, + # which calls a run suffix-shaped when ANY period-chunk is suffix + # VOCABULARY -- and every honorific tail is a suffix word by the + # Lexicon invariant, so a glued honorific is itself the evidence. + # The predicate is circular at THIS call site alone -- not because + # segment asks it any earlier (nothing has peeled at either call) + # but because segment SPENDS the answer differently: it reads the + # run's shape and stops, the answer being the structure, while the + # peel reads the same shape and then decides whether to go strip + # the very honorific that produced it. "이, J.씨" reads as wholly suffix + # only because of the 씨 the peel exists to remove, and declining a + # run that holds the only site does not fall back to some other + # site -- it loses the peel outright, and with it the given name, + # which lands in suffix as "J.씨". Asking for a site in segments[0] + # keeps the #319 answer wherever the peel has somewhere else to go + # ("田中さん, V." still declines, さん is right there) and gives the + # circular case back to master's reading. The two-honorific input + # "김민준씨, J.씨" is where the choice is visible and deliberate: + # both runs offer a site, so the decline stands and the person's own + # 씨 is peeled rather than the junk one behind the comma. + runs = state.segments[:1] + if state.structure is Structure.FAMILY_COMMA: + second = [state.tokens[j].text for j in state.segments[1]] + if not (is_wholly_suffix(second, state.lexicon, state.policy) + and _peel_site(state, state.segments[0], tails)): + runs = state.segments[:2] + site = _peel_site(state, [j for seg in runs for j in seg], tails) + if site is None: return state - text = state.tokens[i].text - # range/cap construction identical to the surname match below, and - # for the same two reasons: longest-first, and a len-1 cap that - # makes the offset interior by construction (_split's contract). - cap = min(_longest_entry(tails), len(text) - 1) - for length in range(cap, 0, -1): - if text[-length:] in tails: - # The tail carries a tag because this stage MANUFACTURED - # it. The segmenter's neighbour test below needs to tell it - # from a token somebody wrote, and no vocabulary question - # can: the two spellings put the same word in the same - # place, and only the provenance differs. - return _split(state, i, (len(text) - length,), None, - tail_tag=_PEELED_TAG) - return state + i, length = site + # The tail carries a tag because this stage MANUFACTURED it. The + # segmenter's neighbour test below needs to tell it from a token + # somebody wrote, and no vocabulary question can: the two spellings + # put the same word in the same place, and only the provenance + # differs. + return _split(state, i, (len(state.tokens[i].text) - length,), None, + tail_tag=_PEELED_TAG) def _split_surname_site(state: ParseState) -> ParseState: diff --git a/nameparser/_pipeline/_segment.py b/nameparser/_pipeline/_segment.py index 6a734f8..4fe3d55 100644 --- a/nameparser/_pipeline/_segment.py +++ b/nameparser/_pipeline/_segment.py @@ -4,19 +4,26 @@ Produces: segments (runs of main-token indices; interior segments may be EMPTY -- doubled commas keep their structural position), structure, COMMA_STRUCTURE ambiguities for unrecognized extra segments. -Reads: Lexicon suffix vocabulary (via _vocab.is_suffix_lenient) -- -the suffix-comma decision is definitionally vocabulary-dependent -(recorded plan deviation #3); reads Policy.lenient_comma_suffixes -to pick the lenient or strict predicate, and -Policy.extra_suffix_delimiters for v1 suffix_delimiter parity (a -delimiter-core token is transparent in the all-suffix tests). +Reads: Lexicon suffix vocabulary and Policy, both through +_vocab.is_wholly_suffix -- the suffix-comma decision is definitionally +vocabulary-dependent (recorded plan deviation #3), and the predicate +owns the rest (Policy.lenient_comma_suffixes picks the lenient or +strict token test; Policy.extra_suffix_delimiters gives v1 +suffix_delimiter parity, a delimiter-core token being transparent). -Decision (v1 parity): >=1 comma and every post-first segment entirely -lenient-suffix AND >1 word before the first comma -> SUFFIX_COMMA; -otherwise FAMILY_COMMA ("Family, Given ..."), with segments beyond the -second that are not lenient-suffix flagged COMMA_STRUCTURE (they are -still best-effort consumed as suffixes by assign, since parse must -stay total over str input and never raise on content). +Decision (v1 parity): >=1 comma and the SECOND segment entirely +suffix AND >1 word before the first comma -> SUFFIX_COMMA; otherwise +FAMILY_COMMA ("Family, Given ..."). Only the second segment decides +(v1 parser.py:1318) -- segments beyond it are consumed as suffixes +either way, and a NON-EMPTY one that is not entirely suffix is +flagged COMMA_STRUCTURE rather than vetoing the structure (an empty +one is consumed silently, as v1 consumed it, so "John Smith, MD,, Jr." +reports nothing where "John Smith, MD, Bart" does; they are still +best-effort consumed as suffixes by assign, since parse must stay +total over str input and never raise on content). "Entirely suffix" +is is_wholly_suffix's question, so the token test inside it is +lenient by default and strict under +Policy(lenient_comma_suffixes=False). """ from __future__ import annotations @@ -24,12 +31,7 @@ import dataclasses from nameparser._pipeline._state import ParseState, PendingAmbiguity, Structure -from nameparser._pipeline._vocab import ( - D as _D, - PH as _PH, - delimiter_cores, is_suffix_lenient, is_suffix_strict, - period_joined_vocab, splits_into_suffixes, -) +from nameparser._pipeline._vocab import is_wholly_suffix from nameparser._types import AmbiguityKind @@ -64,40 +66,9 @@ def segment(state: ParseState) -> ParseState: return dataclasses.replace(state, segments=segs, structure=Structure.NO_COMMA) - # lenient_comma_suffixes=False drops the post-comma test back to - # the strict predicate (initial-shaped suffix words stop qualifying) - predicate = (is_suffix_lenient if state.policy.lenient_comma_suffixes - else is_suffix_strict) - # v1 expand_suffix_delimiter parity (#191): a configured delimiter - # is TRANSPARENT in the all-suffix tests -- v1 split the part string - # on the delimiter before checking, so the delimiter never counted - cores = delimiter_cores(state.policy.extra_suffix_delimiters) - - def counts_as_suffix(text: str) -> bool: - if text in cores: - return True - return (predicate(text, state.lexicon) - or period_joined_vocab(text, state.lexicon) == "suffix" - or (bool(cores) - and splits_into_suffixes(text, cores, state.lexicon))) - def suffixy(seg: tuple[int, ...]) -> bool: - # an EMPTY segment is not suffix-shaped: v1's suffix-comma - # detection fails on an empty parts[1] ('John Smith,, MD' is a - # family-comma parse). An adjacent Ph./D. pair counts as ONE - # suffix unit (v1's fix_phd extracted the credential pre-parse, - # so 'Smith, Ph. D.' read as suffix-comma; keep in sync with - # group's _PH/_D merge). - if not seg: - return False - texts = [state.tokens[i].text for i in seg] - k = 0 - while k < len(texts) - 1: - if _PH.fullmatch(texts[k]) and _D.fullmatch(texts[k + 1]): - texts[k:k + 2] = ["phd"] - else: - k += 1 - return all(counts_as_suffix(t) for t in texts) + return is_wholly_suffix([state.tokens[i].text for i in seg], + state.lexicon, state.policy) # v1 parity: only parts[1] decides the suffix-comma structure # (parser.py:1318); parts[2:] are consumed as suffixes diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 86afd03..159170c 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -4,16 +4,27 @@ predicates live with their stage. All take normalized-or-raw text explicitly -- no state. +is_wholly_suffix departs from that shape twice, deliberately. It is +RUN-level rather than text-level, because the question it answers is +genuinely about a run: the Ph./D. merge spans two tokens, so no +per-token predicate composed with all() can express it. And it takes +the Policy OBJECT, where delimiter_cores takes a pre-extracted +frozenset so its caller hands in one field rather than the config -- +is_wholly_suffix needs TWO policy fields (lenient_comma_suffixes and +extra_suffix_delimiters), and threading both past every caller costs +more than the config parameter saves. Still no state: Policy is frozen +config, not pipeline state. + Layering: imports _lexicon, _types, and _policy only. """ from __future__ import annotations import re import unicodedata -from collections.abc import Callable, Iterable +from collections.abc import Callable, Iterable, Sequence from nameparser._lexicon import Lexicon, _normalize -from nameparser._policy import (Script, _JA_SCRIPTS, _NO_INITIALS, +from nameparser._policy import (Policy, Script, _JA_SCRIPTS, _NO_INITIALS, _SCRIPT_RANGES, _script_matcher) # Ported verbatim from v1 (nameparser/config/regexes.py "initial") minus @@ -37,8 +48,9 @@ _PERIOD_NOT_AT_END = re.compile(r".*\..+$", re.I) # The fix_phd credential pair ('Ph.' + 'D.' as adjacent tokens), shared -# by segment's suffix-comma detection and group's merge (v1 extracted -# the credential pre-parse; the two stages must agree on the pattern). +# by is_wholly_suffix below and group's merge (v1 extracted the +# credential pre-parse; the predicate and the stage must agree on the +# pattern). PH = re.compile(r"^ph\.?$", re.IGNORECASE) D = re.compile(r"^d\.?$", re.IGNORECASE) @@ -166,7 +178,6 @@ def splits_into_suffixes(text: str, cores: frozenset[str], return False - def period_joined_vocab(text: str, lexicon: Lexicon) -> str | None: """v1's parse_pieces derivation for interior-period tokens ('Lt.Gov.', 'Msc.Ed.', and by the ANY rule 'Mr.Smith'): ANY title @@ -186,6 +197,57 @@ def period_joined_vocab(text: str, lexicon: Lexicon) -> str | None: return None +def is_wholly_suffix(texts: Sequence[str], lexicon: Lexicon, + policy: Policy) -> bool: + """Every token in a RUN counts as a suffix -- segment's + suffix-comma test, lifted out of it so the peel can ask the same + question (#319). + + NOT the plural of _script_segment._is_post_nominal, which asks + is_suffix_strict per token. This asks the POLICY-selected predicate + (lenient by default), plus period_joined_vocab, delimiter + transparency and the Ph./D. merge. 'V.' is the input that tells + them apart: it satisfies this predicate but is not a post-nominal + -- and reading one for the other IS the #319 bug. + + An EMPTY run is False, not vacuously True: v1's suffix-comma + detection fails on an empty parts[1] ('John Smith,, MD' is a + family-comma parse). The 'wholly' idiom agrees -- _script_matcher's + whole=True requires non-empty too -- which is why the name is that + one rather than all_suffixes, where Python's all([]) would promise + the opposite. + + An adjacent Ph./D. pair counts as ONE unit (v1's fix_phd extracted + the credential pre-parse, so 'Smith, Ph. D.' read as suffix-comma); + keep in sync with group's _PH/_D merge. + """ + if not texts: + return False + predicate = (is_suffix_lenient if policy.lenient_comma_suffixes + else is_suffix_strict) + # v1 expand_suffix_delimiter parity (#191): a configured delimiter + # is TRANSPARENT in the all-suffix tests -- v1 split the part string + # on the delimiter before checking, so the delimiter never counted + cores = delimiter_cores(policy.extra_suffix_delimiters) + + def counts_as_suffix(text: str) -> bool: + if text in cores: + return True + return (predicate(text, lexicon) + or period_joined_vocab(text, lexicon) == "suffix" + or (bool(cores) + and splits_into_suffixes(text, cores, lexicon))) + + merged = list(texts) + k = 0 + while k < len(merged) - 1: + if PH.fullmatch(merged[k]) and D.fullmatch(merged[k + 1]): + merged[k:k + 2] = ["phd"] + else: + k += 1 + return all(counts_as_suffix(t) for t in merged) + + def _normalized_for_script(text: str) -> str | None: """The guard AND the NFC normalization single_script and effective_script's license path both need, single-sourced so they diff --git a/tests/v2/cases.py b/tests/v2/cases.py index e01e859..dfac720 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -8,6 +8,12 @@ issue or a design-decision slug). No silent expectation edits: changing a row means changing its classification. +"UNDETERMINED" is a fourth value and a TEMPORARY one: a row added +before anyone has run its input against 1.4.0 carries it until the +comparison is made and it resolves to one of the three above. It is +not a standing category, and nothing enforces its removal -- a row +still wearing it is a row whose parity is simply unknown. + The v1 suite's full corpus is extracted into this table by the migration plan (facade runner consumes the same rows); this file seeds it with the pinned battery. @@ -148,7 +154,34 @@ def __post_init__(self) -> None: Case("suffix_comma_split_phd", "John Smith, Ph. D.", {"given": "John", "family": "Smith", "suffix": "Ph. D."}, notes="the adjacent Ph./D. pair counts as one suffix unit in " - "the suffix-comma detection (v1 fix_phd parity)"), + "the suffix-comma detection (v1 fix_phd parity). The " + "pair leads the run here; the row below is where it does " + "not"), + Case("suffix_comma_split_phd_after_another_suffix", + "John Smith, Jr. Ph. D.", + {"given": "John", "family": "Smith", "suffix": "Jr. Ph. D."}, + classification="fix(credential-pair-order)", + notes="the pair is merged wherever it sits in the run, not " + "only at its head, and the row above cannot say so -- " + "there the pair IS the head. Restricted to position 0 " + "the merge never fires, 'D.' is suffix vocabulary in no " + "lexicon, and the run stops being wholly suffix: this " + "reads as a FAMILY comma instead, family 'John Smith' / " + "title 'Jr.' / suffix 'Ph. D.'. Pinned at the segment " + "stage too (test_segment.test_the_credential_pair_" + "merges_anywhere_in_the_run), because the neighbouring " + "input 'John Smith, MD, Jr. Ph. D.' keeps every field " + "under that break and gains only a comma-structure " + "ambiguity, so fields alone do not catch it. Measured: " + "1.4.0 gave suffix 'Ph. D., Jr.' -- fix_phd EXTRACTED " + "the pair pre-parse and re-appended it, reordering the " + "tail, where 2.0 renders it as written. Same words, " + "same roles, different order, and the harness cannot " + "currently see it: expected_changes.toml states that a " + "diffing trailing 'Ph. D.' must fail the run, but this " + "input is absorbed by fix(comma-family), whose " + "name_regex is a bare comma -- measured on a probe " + "corpus, unexplained 0. See the note there"), Case("tail_segment_entry_space_joined", "John Smith, V MD", {"given": "John", "family": "Smith", "suffix": "V MD"}, notes="v1 renders each tail comma segment as ONE suffix " @@ -781,30 +814,67 @@ def __post_init__(self) -> None: "the comma row below, reached without a comma: the peel " "scans segment 0 either way, and with '様.' no longer " "vetoed the scan-back steps over it onto 田中さん and " - "peels さん. Worth its own row because the comma form " - "arrives through a different branch -- FAMILY_COMMA " - "flattens TWO runs before scanning, this one has a " - "single run -- so the comma row can stay green through " - "a change that breaks this one. 1.4.0 read this first " + "peels さん. Worth its own row because it is the only " + "one of the pair where the SCAN-BACK is what #320 " + "fixes, and since #319 the only one where the PEEL " + "depends on #320 at all: here '様.' shares the run with " + "田中さん and has to be stepped over, while the comma " + "form declines its post-comma run and never looks at " + "'様.' in the peel. That decline does not carry the " + "veto over -- is_wholly_suffix asks is_suffix_lenient " + "under the default policy, which takes '様.' whether " + "the veto is in place or not (see the row below) -- so " + "restoring the veto strands さん HERE and nowhere else " + "in the pair, giving given '田中さん', family '様.'. " + "(Before #319 the distinction was structural -- " + "FAMILY_COMMA flattened TWO runs before scanning where " + "this has one -- which is no longer what separates " + "them.) 1.4.0 read this first " "田中さん / last '様.', which is what 2.0 produced until " "#320: parity before, a classified change after"), Case("ja_honorific_period_does_not_stop_the_peel", "田中さん, 様.", {"family": "田中", "suffix": "さん, 様."}, classification="fix(#320)", - notes="#320's real cost: the veto did not merely misfile " - "'様.' -- it made _is_post_nominal say no, so the #312 " - "peel's scan-back stopped AT '様.' as the site instead " - "of stepping over it, found no listed tail there, and " - "abandoned the peel. One period and さん stayed glued " - "to 田中. Exactly the shape of " + notes="the comma spelling of the row above, and since #319 a " + "DIFFERENT dependence on #320. At #320 this row was " + "about the peel: the veto made _is_suffix_strict say " + "no of '様.', so the #312 scan-back stopped AT it as " + "the site, found no listed tail there and abandoned, " + "leaving さん glued to 田中 over one period. #319 then " + "declined the post-comma run outright, and what is " + "left here for #320 to fix is the ASSIGNMENT -- with " + "the veto restored the peel still fires (family 田中, " + "suffix さん) and only '様.' moves, to given. The " + "fields and the classification are unchanged either " + "way. CORRECTION, recorded here because commit " + "4aff219's message claims otherwise and cannot be " + "amended: the veto does NOT reach is_wholly_suffix " + "under this row's DEFAULT policy, so it is not true " + "that #319 merely handed the same veto to a different " + "predicate. is_wholly_suffix selects is_suffix_lenient " + "there, whose contract is suffix_words accepted " + "unconditionally, BYPASSING the initial veto, and " + "_normalize('様.') is '様', a suffix word -- so " + "is_wholly_suffix(['様.']) is True veto or no veto, " + "the run is declined either way, and simulating the " + "veto PEELS さん here rather than stranding it. The " + "is_wholly_suffix route to an abandoned peel exists " + "only under Policy(lenient_comma_suffixes=False), " + "which drops the call to is_suffix_strict and gives " + "family '田中さん', given '様.'; this row does not set " + "that knob, and " + "ko_honorific_period_under_strict_comma_suffixes is " + "the table's row for it. Was the " + "shape of " "ja_honorific_glued_before_an_initial above, except " "that here the token stopping the scan is a real " "honorific rather than an initial, which is what makes " "it a bug rather than the intended veto. The structure " - "is FAMILY_COMMA before and after -- SUFFIX_COMMA needs " - "more than one word ahead of the comma and 田中さん is " - "one -- so the two runs were always in the peel's " - "reach; only the strict test's answer moved. 1.4.0 read " + "is FAMILY_COMMA throughout -- SUFFIX_COMMA needs more " + "than one word ahead of the comma and 田中さん is one -- " + "so at #320 both runs were in the peel's reach and " + "only the strict test's answer moved. " + "1.4.0 read " "this first '様.' / last 田中さん, which is exactly what " "2.0 produced before this change -- the row sat at " "parity until #320 moved it"), @@ -841,7 +911,12 @@ def __post_init__(self) -> None: policy=Policy(lenient_comma_suffixes=False), classification="fix(#320)", notes="the row above under the knob that governs exactly this " - "shape, and the table's only exercise of it. " + "shape, and one of the table's three exercises of it " + "(ja_honorific_glued_family_comma_strict_knob is the " + "second, on the initial-shaped side of the same gap; " + "ja_honorific_glued_family_comma_credential_pair_strict_" + "knob is the third, showing where the knob changes " + "nothing). " "lenient_comma_suffixes=False drops segment's post-comma " "test to the strict one, so a 'Family, Suffix' input " "whose suffix is INITIAL-SHAPED reads as a given-name " @@ -852,9 +927,11 @@ def __post_init__(self) -> None: "and the honorific parses identically under both " "settings -- which is the claim this row exists to " "hold, since the knob's own documentation scopes it to " - "the roman numerals 'I' and 'V' and _script_segment " - "names it as the setting that strands 씨 in a " - "neighbouring shape. No v1 spelling exists for the knob " + "the initial-shaped suffix words ('John Smith, V') and " + "_script_segment names it as the setting that keeps a " + "glued honorific inside the name in a neighbouring " + "shape ('田中さん, V.' gives family '田中さん' under " + "it). No v1 spelling exists for the knob " "(the facade runner skips this row), so the " "classification compares against 1.4.0's single " "reading, first '씨.' / last 김민준 -- the same fields " @@ -1075,10 +1152,20 @@ def __post_init__(self) -> None: Case("ja_honorific_glued_family_comma", "田中さん, PhD", {"title": "PhD", "family": "田中", "suffix": "さん"}, classification="fix(#312)", - notes="the peel crosses the comma: PhD is a post-nominal, so " - "the site scan steps over it and reaches 田中さん. " - "Agrees with the spaced 田中さん PhD, which peels for " - "the same reason. Where PhD itself lands still differs " + notes="the peel reaches 田中さん across the comma, though no " + "longer by crossing it: since #319 is_wholly_suffix " + "declines the post-comma run outright (PhD is suffix " + "vocabulary and it is the whole run), so the scan never " + "leaves segments[0] and never examines PhD at all. It " + "did cross before, stepping OVER PhD because that " + "spelling satisfies _is_post_nominal's strict test -- " + "the same fields by the older route. The spaced " + "田中さん PhD still peels that way, its single run " + "holding both tokens, so the two spellings now agree on " + "the outcome through DIFFERENT mechanisms; " + "ja_honorific_glued_family_comma_suffixy_second_run " + "cites this row for the outcome, not the route. Where " + "PhD itself lands still differs " "between the two spellings -- suffix spaced, title " "post-comma -- and that is fix(comma-family)'s, not " "the peel's. The expectation bakes in TWO deviations " @@ -1089,25 +1176,223 @@ def __post_init__(self) -> None: "rule is reaching that half"), Case("ja_honorific_glued_family_comma_suffixy_second_run", "田中さん, V.", + {"given": "V.", "family": "田中", "suffix": "さん"}, + classification="fix(#319)", + notes="under a family comma the peel scanned both runs on " + "the premise that segments[1] is name text, and here it " + "is not: segment picks FAMILY_COMMA when the pre-comma " + "part is a single word, even where the post-comma part " + "is entirely suffix-shaped, so the scan reached 'V.' -- " + "which is_suffix_strict rejects as an initial where " + "segment admitted the run on is_suffix_lenient. 'V.' " + "was therefore the site, ended in no tail, and the peel " + "was abandoned with さん still in the family name. #319 " + "asks segment's own predicate (_vocab.is_wholly_suffix) " + "instead of inferring name text from the structure: the " + "run is declined, the scan stays inside segments[0], " + "and さん peels off 田中さん as it always did without a " + "second run. Same credential, three spellings, ONE " + "answer FROM THE PEEL now -- this, '田中さん, PhD' " + "above and '田中さん, Ph. D.' in " + "ja_honorific_glued_family_comma_credential_pair below " + "all reach 田中さん. Where the credential itself LANDS " + "still differs by spelling (title 'PhD', given 'V.', " + "suffix 'さん, Ph. D.'); that is assign's question, not the " + "peel's, and this row is not a claim about it. Nor does " + "the comma form now agree with its SPACED twin: " + "ja_honorific_glued_before_an_initial ('田中さん V.') " + "still does not peel, because under NO_COMMA there is " + "one run and 'V.' is inside the name's own tokens where " + "the scan-back legitimately stops -- declining a " + "post-comma run does not reach it. Parity with 1.4.0 " + "(first V., last 田中さん) until this change, which is " + "what moves it; the 1.4.0 fields are still reachable " + "through Policy(lenient_comma_suffixes=False), pinned " + "by ja_honorific_glued_family_comma_strict_knob below"), + Case("ja_honorific_glued_family_comma_credential_pair", + "田中さん, Ph. D.", + {"family": "田中", "suffix": "さん, Ph. D."}, + classification="fix(#319)", + notes="the third of the three spellings #319 named, and the " + "only one the Ph./D. merge reaches: is_wholly_suffix " + "folds the adjacent pair into the single unit 'phd' " + "(v1's fix_phd extracted the credential pre-parse), so " + "the run counts as wholly suffix, the peel declines it, " + "and さん comes off 田中さん exactly as in " + "ja_honorific_glued_family_comma_suffixy_second_run " + "above. The merge is also why this spelling is NOT the " + "one to reach for when exercising " + "Policy(lenient_comma_suffixes=False): 'phd' satisfies " + "is_suffix_strict as readily as is_suffix_lenient, so " + "the run is declined under EITHER setting and this row " + "is identical under the knob, which " + "ja_honorific_glued_family_comma_credential_pair_strict_" + "knob below holds rather than leaving to the claim -- " + "ja_honorific_glued_family_comma_strict_knob below uses " + "'V.' because the initial-shaped words are where the " + "two predicates actually part. Where the credential " + "lands is a separate question and answers differently " + "again: suffix here, title for 'PhD', given for 'V.' " + "Measured: 1.4.0 gave first 田中さん / suffix 'Ph. D.' " + "(fix_phd lifted the credential pre-parse, leaving a " + "lone pre-comma word), so like " + "ja_honorific_glued_family_comma above the expectation " + "carries TWO deviations -- the peel is #319's, first -> " + "family is comma-family's, which 2.0 already had before " + "this change (family 田中さん / suffix 'Ph. D.')"), + Case("ja_honorific_glued_family_comma_strict_knob", "田中さん, V.", {"family": "田中さん", "given": "V."}, - notes="a KNOWN LIMIT, recorded as it behaves rather than " - "fixed. Under a family comma the peel scans both runs " - "on the premise that segments[1] is name text, and here " - "it is not: segment picks FAMILY_COMMA when the " - "pre-comma part is a single word, even where the " - "post-comma part is entirely suffix-shaped, so the scan " - "reaches 'V.' -- which is_suffix_strict rejects as an " - "initial where segment admitted the run on " - "is_suffix_lenient. 'V.' is therefore the site, ends in " - "no tail, and the peel is abandoned with さん still in " - "the family name. Same credential, three spellings, two " - "answers: '田中さん, PhD' peels (above) while this and " - "'田中さん, Ph. D.' do not. Parity with 1.4.0 (first " - "V., last 田中さん) and unchanged by #312 -- the " - "strict/lenient gap predates it, and closing it wants " - "segment's own suffixy test extracted into a shared " - "predicate, which is why the limit is stated here " - "instead of fixed"), + policy=Policy(lenient_comma_suffixes=False), + classification="parity", + notes="the same input as " + "ja_honorific_glued_family_comma_suffixy_second_run " + "above under the knob that keeps the pre-#319 answer, " + "and one of the table's three exercises of it " + "(ko_honorific_period_under_strict_comma_suffixes and " + "ja_honorific_glued_family_comma_credential_pair_strict_" + "knob are the others). The knob drops is_wholly_suffix " + "to the " + "strict predicate, which rejects 'V.' as an initial, so " + "the post-comma run reads as name text after all, the " + "scan crosses into it, 'V.' is the site, it ends in no " + "listed tail and the peel is abandoned -- さん stays in " + "the family name. Not a blanket freeze of pre-#319 " + "behavior, and the row must not be read as one: it " + "holds for the INITIAL-shaped suffix words ('V.', 'V', " + "'I'), which is the whole of where the strict/lenient " + "gap lives. The counterexample is the row above, " + "ja_honorific_glued_family_comma_credential_pair: its " + "Ph./D. pair merges to a form strict accepts too, so " + "that run is " + "declined and the peel fires under this setting as " + "well -- pinned, not merely stated, by " + "ja_honorific_glued_family_comma_credential_pair_strict_" + "knob above. No v1 spelling exists for the knob, so the " + "facade runner skips this row and the classification " + "compares against 1.4.0's single reading of the same " + "text, as the other exercise of the knob named above " + "does: measured, 1.4.0 gave first 'V.' / last " + "田中さん, which is field for field what the knob holds " + "here -- parity, and the point of the knob"), + Case("ja_honorific_glued_family_comma_credential_pair_strict_knob", + "田中さん, Ph. D.", + {"family": "田中", "suffix": "さん, Ph. D."}, + policy=Policy(lenient_comma_suffixes=False), + classification="fix(#319)", + notes="the counterexample the two rows above assert and " + "neither measured: the knob does NOT freeze the " + "pre-#319 reading in general, only for the " + "initial-shaped suffix words. Here the Ph./D. merge " + "folds the run into 'phd', which is_suffix_strict " + "accepts as readily as is_suffix_lenient, so the run is " + "declined and さん peels under this setting exactly as " + "under the default -- field for field the same " + "expectation as " + "ja_honorific_glued_family_comma_credential_pair, which " + "is the whole claim. Cheap to hold and worth holding " + "separately, because the two rows differ only in the " + "policy and a knob that started gating the decline " + "wholesale would move this one alone. Same " + "classification and the same two deviations as its " + "default-policy twin (1.4.0: first 田中さん / suffix " + "'Ph. D.'), which is also why the knob cannot be judged " + "against a v1 spelling here -- there is none, so the " + "facade runner skips this row as it does the other two " + "knob rows"), + Case("ko_honorific_glued_family_comma_suffixy_second_run", + "김민준씨, V.", + {"given": "V.", "family": "김민준", "suffix": "씨"}, + classification="fix(#319)", + notes="ja_honorific_glued_family_comma_suffixy_second_run in " + "hangul, and pinned because the decline reads no script " + "at all: it asks is_wholly_suffix about the post-comma " + "run and the peel's own scan-back about segments[0], " + "both vocabulary questions, so a script-conditional " + "regression would be invisible in a table whose every " + "other witness to #319 is written in kana. The family " + "stays 김민준 undivided -- the FAMILY comma gates the " + "surname split off, so hangul segmentation never runs " + "here and only the peel acts. 1.4.0 gave first 'V.' / " + "last 김민준씨, peeling nothing"), + Case("zh_honorific_glued_family_comma_suffixy_second_run", + "王先生, V.", + {"given": "V.", "family": "王", "suffix": "先生"}, + classification="fix(#319)", + notes="the Han spelling of the row above, and the third " + "script. 先生 is a shipped tail, so the peel fires with " + "no locale opted in -- honorific_tails is licensed by " + "the entries themselves rather than by " + "Policy.segment_scripts, and HAN is not activated here " + "(the family is what the peel left behind, not a " + "vocabulary split). 1.4.0 gave first 'V.' / last " + "王先生"), + Case("ko_honorific_glued_family_comma_site_only_beyond_the_comma", + "이, J.씨", + {"given": "J.", "family": "이", "suffix": "씨"}, + classification="fix(#312)", + notes="the limit of #319's decline, and the row that says why " + "it carries a second condition. is_wholly_suffix reaches " + "period_joined_vocab, which calls an interior-period " + "token a suffix when ANY chunk is suffix vocabulary -- " + "and every honorific tail is a suffix WORD by the " + "Lexicon invariant, so 'J.씨' reads as suffix-shaped " + "BECAUSE of the 씨 the peel exists to remove. Declining " + "on that evidence does not move the peel elsewhere the " + "way 田中さん, V. does: segments[0] is the lone 이, which " + "ends in no tail, so the scan finds no site at all, " + "nothing peels, and the given name goes to suffix " + "glued to its honorific ('J.씨'). So the gate declines " + "only where segments[0] holds a peel site of its own, " + "and this input reaches the pre-#319 fields by having " + "none. Not rescued by " + "Policy(lenient_comma_suffixes=False) either, unlike the " + "strict-knob row above: the knob picks between " + "is_suffix_lenient and is_suffix_strict per token and " + "period_joined_vocab is downstream of neither, so both " + "settings call this run wholly suffix. 1.4.0 gave first " + "'J.씨' / last 이 -- it peels nothing, so the deviation " + "here is #312's crossing, which is what puts the site on " + "'J.씨' in the first place"), + Case("ko_honorific_glued_family_comma_site_in_both_runs", + "김민준씨, J.씨", + {"family": "김민준", "suffix": "씨, J.씨"}, + classification="fix(#319)", + notes="the two-honorific input, where both runs hold a site " + "and the decline therefore stands -- a deliberate choice " + "between two readings rather than a fallout. Pre-#319 " + "the scan crossed and took the LAST site, peeling the 씨 " + "off the junk 'J.씨' and leaving the person's own " + "honorific glued in the family name (given 'J.', family " + "김민준씨, suffix 씨); now 씨 comes off 김민준씨 and " + "'J.씨' is consumed whole as a suffix. It is also the " + "row that rules out the narrower gate: declining only " + "where the SECOND run has no peel site fixes " + "ko_honorific_glued_family_comma_site_only_beyond_the_" + "comma above and reverts this input to the pre-#319 " + "reading, which is the worse of the two -- the same " + "junk-tail reach " + "ko_honorific_glued_given_suffix_comma_initial's note " + "names under a suffix comma. 1.4.0 gave first 'J.씨' / " + "last 김민준씨, peeling neither"), + Case("ko_honorific_glued_family_comma_lone_post_nominal_before_it", + "선생님, J.씨", + {"given": "J.", "family": "선생님", "suffix": "씨"}, + classification="fix(#312)", + notes="segments[0] is a single token that IS a listed tail " + "entire, which the site scan skips as a post-nominal " + "(the same guard that keeps 선생님 from being peeled to " + "선생 + 님) -- so there is no site before the comma, the " + "run beyond it is scanned after all, and the fields are " + "the pre-#319 ones as in the 이 row above. Pinned " + "because it separates asking for the site with the " + "peel's own scan-back from asking the cheaper question, " + "whether any token ENDS in a tail: the cheaper one " + "counts 선생님, declines, and then finds nothing to cut " + "-- 씨 lost into suffix 'J.씨'. Only the lone-token " + "shape of that divergence is reachable from the gate, " + "which is why this row is one token before the comma; " + "_peel_site's docstring derives the bound. " + "1.4.0 gave first 'J.씨' / last 선생님"), Case("ko_honorific_glued_given_after_family_comma", "김, 민준씨", {"family": "김", "given": "민준", "suffix": "씨"}, classification="fix(#312)", @@ -1148,7 +1433,17 @@ def __post_init__(self) -> None: "while the site scan asks is_suffix_strict and an " "initial fails it. 'V.' then ends in no tail, the peel " "is abandoned, and 씨 is back in the given name -- the " - "original bug, and no other row in this table notices. " + "original bug. It was the table's only witness to that " + "widening until #319; ja_honorific_glued_family_comma_" + "suffixy_second_run and " + "ja_honorific_glued_family_comma_credential_pair notice " + "it as well now, but from the FAMILY_COMMA side, where " + "the guard is is_wholly_suffix rather than this " + "structural scoping. Their strict-knob sibling does NOT " + "join them: under the knob is_wholly_suffix is False on " + "the same run, so the scan was already crossing into " + "segments[1] and widening past it changes nothing. This " + "row is still the only SUFFIX comma among the three. " "Its comma-less twin ja_honorific_glued_before_an_initial " "shows the same veto from the other side, where 'V.' is " "in the name's own run and so IS the site"), diff --git a/tests/v2/pipeline/test_script_segment.py b/tests/v2/pipeline/test_script_segment.py index 5cab446..8db290a 100644 --- a/tests/v2/pipeline/test_script_segment.py +++ b/tests/v2/pipeline/test_script_segment.py @@ -594,6 +594,21 @@ def test_the_peel_scans_the_name_runs_and_no_further() -> None: "Dr", "김", "민준", "씨", "Jr.", "박씨"] +def test_a_suffix_comma_keeps_the_peel_inside_the_name() -> None: + # The test above shows a suffix comma not being crossed while the + # name HOLDS a site, which is the easy half: segments[0] answers + # the scan either way, so a gate that had stopped keying on + # FAMILY_COMMA -- on "there is a second run at all", say -- would + # still land the peel on 민준씨 and pass. This is the twin that can + # see it: two words before the comma so the structure is + # SUFFIX_COMMA, NO site among them, and a token beyond the comma + # that ends in a listed tail. The peel must answer None, not reach + # past the name for the junk one -- "J.씨" is a stray post-nominal, + # and its 씨 is nobody's honorific. + assert _texts(_run("John Smith, J.씨", policy=_HANGUL, + lexicon=_LEX_TAILS)) == ["John", "Smith", "J.씨"] + + def test_the_peel_crosses_a_family_comma_and_stops_there() -> None: # The family-comma mirror of the test above, and the only input # that pins the SECOND half of segments[:2] -- that it is two runs @@ -601,16 +616,96 @@ def test_the_peel_crosses_a_family_comma_and_stops_there() -> None: # two segments, so the slice is otherwise only ever exercised as # "more than one run": widening it to three passes them all. # Both shapes above, re-spelled with the name split across a family - # comma. "Jr." is the strict/lenient gap again -- reached as a site - # it ends in no tail and abandons the peel, leaving 씨 in the given - # name -- and 박씨 is the junk tail, whose 씨 a wider scan peels - # instead of the person's own. + # comma -- but only the SECOND of the two pins the slice, and the + # measurement says so: widen this scan to three runs and 박씨 is + # the junk tail whose 씨 gets peeled instead of the person's own, + # while "Jr." is a post-nominal by the STRICT test as well, so a + # wider scan steps over it onto 민준씨 and peels correctly anyway. + # It is the initial-shaped words ("V." in the test above) that fall + # in the strict/lenient gap, not "Jr."; this input keeps 씨's + # placement pinned against a third run of ordinary post-nominals. assert _texts(_run("김, 민준씨, Jr.", policy=_HANGUL, lexicon=_LEX_TAILS)) == ["김", "민준", "씨", "Jr."] assert _texts(_run("김, 민준씨, 박씨", policy=_HANGUL, lexicon=_LEX_TAILS)) == ["김", "민준", "씨", "박씨"] +def test_a_wholly_suffix_run_after_a_family_comma_is_declined() -> None: + # The two tests above pin HOW FAR the crossing reaches; this pins + # WHETHER it happens, which the structure alone does not decide. + # segment answers FAMILY_COMMA whenever a single word precedes the + # comma, even where the part after it is entirely suffix-shaped, so + # "the second run is name text" is an inference and a wrong one + # here. Scanning it lands the site on "V." -- admitted to the run by + # segment's is_suffix_lenient, rejected as an initial by the scan's + # is_suffix_strict -- which ends in no listed tail, so the peel is + # abandoned and さん stays glued to 田中 (#319). The peel asks + # is_wholly_suffix, segment's own predicate, and declines the run. + lex = _LEX_TAILS.add(suffix_words={"v"}) + assert _texts(_run("田中さん, V.", policy=_HANGUL, + lexicon=lex)) == ["田中", "さん", "V."] + + +def test_a_long_wholly_suffix_post_comma_run_is_declined_too() -> None: + # The decline is a question about the run's VOCABULARY and not its + # LENGTH, and no other input here says so: every case pinning the + # decline has a one- or two-token run, so a gate that declined only + # short runs passes all of them and this one alone catches it. + # Three post-nominals, with the initial-shaped one LEFTMOST on + # purpose -- a scan that reached this run would step over PhD and + # MD (both post-nominals by the strict test) and stop on "V.", + # which ends in no tail, abandoning the peel exactly as above. + lex = _LEX_TAILS.add(suffix_words={"v"}, + suffix_acronyms={"md", "phd"}) + assert _texts(_run("田中さん, V. MD PhD", policy=_HANGUL, + lexicon=lex)) == ["田中", "さん", "V.", "MD", "PhD"] + + +def test_a_third_segment_does_not_reopen_the_declined_run() -> None: + # WHICH run the decline reads, and how many there may be, are two + # more axes no other input here moves: every case pinning the + # decline has exactly two segments, so reading segments[-1] instead + # of segments[1], or requiring exactly two segments before + # declining at all, passes all of them. Both mistakes give the same + # visible loss on this input -- the run consulted is 太郎, or none + # is, so the scan crosses, "V." is the site, it ends in no tail and + # the peel is abandoned. The third segment is junk beyond the name + # either way (segment flags it, and the peel already refuses to + # scan it -- test_the_peel_crosses_a_family_comma_and_stops_there + # above); the question here is only whether its presence changes + # the answer about the SECOND. + lex = _LEX_TAILS.add(suffix_words={"v"}) + assert _texts(_run("田中さん, V., 太郎", policy=_HANGUL, + lexicon=lex)) == ["田中", "さん", "V.", "太郎"] + + +def test_a_configured_suffix_delimiter_reaches_the_peel() -> None: + # is_wholly_suffix is handed the CALLER's policy, so + # extra_suffix_delimiters decides what counts as a suffix run here + # exactly as it does inside segment -- a peel that passed a default + # policy instead would answer differently from the stage whose + # question it is borrowing. Two routes into that answer, neither + # reachable from the other's input, so both are pinned: a + # delimiter WITHOUT whitespace leaves one token no core equals + # ("PhD/MD"), admitted by splits_into_suffixes, while a + # whitespace-padded one surfaces as a standalone token ("-"), + # admitted by plain core membership. Each peels only with its + # delimiter configured; the bare-policy line under each is the + # baseline that says so. + lex = _LEX_TAILS.add(suffix_words={"v"}, + suffix_acronyms={"md", "phd"}) + slash = Policy(extra_suffix_delimiters=frozenset({"/"})) + assert _texts(_run("田中さん, PhD/MD", policy=slash, + lexicon=lex)) == ["田中", "さん", "PhD/MD"] + assert _texts(_run("田中さん, PhD/MD", policy=_HANGUL, + lexicon=lex)) == ["田中さん", "PhD/MD"] + dash = Policy(extra_suffix_delimiters=frozenset({" - "})) + assert _texts(_run("田中さん, Jr. - V.", policy=dash, + lexicon=lex)) == ["田中", "さん", "Jr.", "-", "V."] + assert _texts(_run("田中さん, Jr. - V.", policy=_HANGUL, + lexicon=lex)) == ["田中さん", "Jr.", "-", "V."] + + def test_an_empty_first_run_still_reaches_the_second() -> None: # A leading comma empties segments[0] without emptying the name: # the scan flattens both runs, so an empty first one is nothing to diff --git a/tests/v2/pipeline/test_segment.py b/tests/v2/pipeline/test_segment.py index 3bf8de8..cd51fa2 100644 --- a/tests/v2/pipeline/test_segment.py +++ b/tests/v2/pipeline/test_segment.py @@ -49,6 +49,20 @@ def test_suffix_comma_lenient_accepts_initial_shaped_suffix_word() -> None: assert out.structure is Structure.SUFFIX_COMMA +def test_the_credential_pair_merges_anywhere_in_the_run() -> None: + # v1's fix_phd healed a split 'Ph. D.' wherever it fell, and the + # suffix-comma test inherits that: the pair counts as ONE unit at + # any position, not just at the head of the run. Nothing else + # reaches past position 0 -- 'John Smith, Ph. D.' above puts the + # pair first -- and the flip is silent, because 'D.' alone is + # suffix vocabulary in no lexicon, so the run stops being wholly + # suffix and this reads as a FAMILY comma instead. + out = _segmented("John Smith, Jr. Ph. D.") + assert out.structure is Structure.SUFFIX_COMMA + assert [_texts(out, s) for s in out.segments] == [ + ["John", "Smith"], ["Jr.", "Ph.", "D."]] + + def test_family_comma_with_trailing_suffix_segment() -> None: out = _segmented("Smith, John, Jr.") assert out.structure is Structure.FAMILY_COMMA diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index b27a21f..3ba574d 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -3,9 +3,10 @@ from nameparser._lexicon import Lexicon from nameparser._pipeline._vocab import ( effective_script, is_initial, is_initial_shaped, is_suffix_lenient, - is_suffix_strict, resolve_script_set, single_script, + is_suffix_strict, is_wholly_suffix, resolve_script_set, single_script, ) -from nameparser._policy import Script, _NO_INITIALS, _SCRIPT_RANGES +from nameparser._policy import (Policy, Script, _NO_INITIALS, + _SCRIPT_RANGES) _LEX = Lexicon( suffix_acronyms=frozenset({"phd", "ma"}), @@ -162,6 +163,65 @@ def test_strict_excludes_bare_ambiguous_even_when_in_acronyms() -> None: assert is_suffix_strict("M.A.", _LEX) +_SUFFIX_LEX = Lexicon( + suffix_acronyms=frozenset({"phd", "md"}), + suffix_words=frozenset({"jr", "v"}), +) + + +def test_is_wholly_suffix() -> None: + lex, pol = _SUFFIX_LEX, Policy() + assert is_wholly_suffix(["Jr."], lex, pol) + assert is_wholly_suffix(["PhD", "Jr."], lex, pol) + assert not is_wholly_suffix(["Smith"], lex, pol) + assert not is_wholly_suffix(["PhD", "Smith"], lex, pol) + # an adjacent Ph./D. pair is ONE unit. 'D.' is what makes the merge + # load-bearing: it is not suffix vocabulary in ANY lexicon, so the + # pair fails without the merge. ('Ph.' happens to fail alone here + # too, but only because _SUFFIX_LEX is synthetic -- under + # Lexicon.default(), 'ph' IS in suffix_acronyms.) + assert is_wholly_suffix(["Ph.", "D."], lex, pol) + assert not is_wholly_suffix(["Ph."], lex, pol) + assert is_wholly_suffix(["Ph.", "D.", "Jr."], lex, pol) + # and the pair is found wherever it sits, not only at the head of + # the run: the merge walks the whole list. 'John Smith, Jr. Ph. D.' + # is the reachable input -- restricted to position 0 the pair never + # merges, 'D.' fails alone, and segment reads a FAMILY comma where + # it should read a suffix comma. + assert is_wholly_suffix(["Jr.", "Ph.", "D."], lex, pol) + # the two routes that are neither the policy-selected predicate nor + # the merge, each with the input that reaches ONLY it. Both die in + # the v1 banks and the case table today, so a break here is + # diagnosed a long way from its cause. + assert is_wholly_suffix(["Lt.Jr."], lex, pol) # period-joined + assert not is_wholly_suffix(["Lt.Smith"], lex, pol) # no suffix chunk + delim = Policy(extra_suffix_delimiters=frozenset({"/"})) + assert is_wholly_suffix(["PhD/MD"], lex, delim) # delimiter split + assert not is_wholly_suffix(["PhD/MD"], lex, pol) + + +def test_is_wholly_suffix_empty_run_is_false() -> None: + """NOT vacuous truth, unlike Python's all() and unlike v1's + are_suffixes. v1's suffix-comma detection fails on an empty + parts[1] -- 'John Smith,, MD' is a family-comma parse -- and the + 'wholly' idiom agrees (_script_matcher(whole=True) requires + non-empty too).""" + assert not is_wholly_suffix([], _SUFFIX_LEX, Policy()) + + +def test_is_wholly_suffix_is_not_the_plural_of_is_post_nominal() -> None: + """The #319 bug in one assertion. _script_segment._is_post_nominal + asks is_suffix_strict per token; this asks the POLICY-selected + predicate over a run. 'V.' is the input that tells them apart: a + suffix run, but not a post-nominal.""" + lex, pol = _SUFFIX_LEX, Policy() + assert is_wholly_suffix(["V."], lex, pol) + assert not is_suffix_strict("V.", lex) + # and the knob moves it: under strict, 'V.' is name text + assert not is_wholly_suffix( + ["V."], lex, Policy(lenient_comma_suffixes=False)) + + def test_single_script_requires_every_char_in_one_script() -> None: assert single_script("毛泽东") is Script.HAN assert single_script("諸葛") is Script.HAN # traditional diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index 357a4d9..a294bb1 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -30,6 +30,7 @@ "毛泽东" "王·Smith" "王先生" +"王先生, V." "王君" "王小明 先生" "王小明先生" @@ -43,6 +44,7 @@ "田中さん II" "田中さん V." "田中さん 様." +"田中さん, Ph. D." "田中さん, PhD" "田中さん, V." "田中さん, 太郎" @@ -75,6 +77,8 @@ "김민준씨" "김민준씨 (Jimmy)" "김민준씨 Jr." +"김민준씨, J.씨" +"김민준씨, V." "김선생님" "김씨" "김지양" @@ -83,6 +87,8 @@ "남궁민수, 지훈" "마이클·잭슨" "선생님" +"선생님, J.씨" "씨" "양 미선" "양 지훈" +"이, J.씨" diff --git a/tools/differential/expected_changes.toml b/tools/differential/expected_changes.toml index 9e4fef9..edb72eb 100644 --- a/tools/differential/expected_changes.toml +++ b/tools/differential/expected_changes.toml @@ -309,3 +309,24 @@ fields = ["title", "first", "middle", "suffix"] # intentionally left unclassified: if it ever starts diffing, the # harness must fail. (The leading case above is a separate shape, which # is why it needed its own anchored rule rather than a widened one.) +# +# That last sentence does NOT hold today, and the scope of what is +# parity has to be read narrowly because of it. Parity is the two +# inputs named above, where the pair LEADS its run. A trailing pair +# after another suffix diffs on ORDER -- 'John Smith, Jr. Ph. D.' is +# 1.4.0 suffix 'Ph. D., Jr.' against 2.0's 'Jr. Ph. D.', because +# fix_phd extracted the credential pre-parse and re-appended it while +# 2.0 renders the tail as written (tests/v2/cases.py +# 'suffix_comma_split_phd_after_another_suffix', +# classification="fix(credential-pair-order)"). Measured on a probe +# corpus: the run comes out unexplained 0, absorbed by +# fix(comma-family) below, whose name_regex is a bare comma and whose +# `fields` list contains `suffix`. So the shape is guarded by the case +# table and not by this file. Left that way deliberately rather than +# fixed in passing: classify() takes the FIRST matching rule and the +# sort has only two tiers (name_regex before fields-only, stable +# within a tier), so a narrower rule for this shape could only win by +# being written earlier in the file -- making file order load-bearing +# again, which is the thing the sort exists to prevent. Giving +# classify() a real specificity order is the fix, and it is a change +# to the harness rather than to a rule.