Stop is_initial matching CJK characters (#320) - #321
Merged
Conversation
An initial is a single letter standing in for a name. v1's regex used
\w, which is Unicode-aware, so a Han ideograph or hangul syllable plus
a period matched -- and is_suffix_strict applies is_initial as a veto,
which pushed period-written CJK honorifics ('씨.', '様.') out of the
suffix vocabulary entirely.
Split the shape test from the repertoire test rather than narrowing
the character class. _INITIAL stays v1-verbatim, so its three pinned
copies (config, _vocab, _render) stay byte-identical; _NO_INITIALS
names the scripts whose characters cannot be an initial and is_initial
composes the two. Narrowing to [A-Za-z] would have been wrong in the
other direction -- it flips 'Й.' from initial to conjunction, the
regression the 2.1.0 Ukrainian entry claims to prevent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adding a Script member already trips a chain of gates: test_policy's test_every_script_member_has_a_range_entry demands a codepoint range, and satisfying that trips test_regex_sync's two differential-rule gates. None of them asks whether the script has initials, which is a separate question from whether it determines an order or a segmentation -- and #317 (Thai) is a live candidate that would otherwise inherit the CJK answer silently. Every gate in that chain is satisfied by editing _SCRIPT_RANGES, the toml's character classes, or the four-member roster literal test_regex_sync spells out -- none of which requires answering the initials question. This one compares a hand-written classification table against set(Script), so it fires at the first step and stays failing through the rest. Verified by adding a temporary Script.THAI member -- 2 failed (this gate and test_policy); adding a range entry too -- 3 failed (this gate and both regex_sync gates). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Cyrillic assertion is the load-bearing one: it asserts the TAG, because the naive narrowing changes 'Й.' from initial to conjunction without moving field output on a short name, which is how it passed a full green suite unnoticed. The collision is live, not hypothetical -- й ships as a default conjunction (Ukrainian, 60943ed for #267) -- so the test reads й from the module's _LEX rather than from a lexicon built in the test body, which would keep passing if the shipped entry were dropped. Two of the three case notes state a different mechanism than the one this change was expected to have. Measured rather than assumed: - '김민준 씨.' does not turn on segmentation, which divides 김민준 identically either way. effective_script('씨.') is None -- the trailing period defeats the wholly-one-script test -- so with '씨.' still a name piece the script_orders lookup declined for the whole name and the pieces fell back to name_order. - '田中さん, 様.' is FAMILY_COMMA before and after; SUFFIX_COMMA needs more than one word ahead of the comma. Both runs were always in the #312 peel's reach. What moved is is_suffix_strict's answer: the peel scan-back stopped at '様.' as the site rather than stepping over it. Each Korean row names the period-free sibling whose fields it must match and before this did not (ko_honorific_after_comma and ko_honorific_ssi); the Japanese row names ja_honorific_glued_before_an_initial, the intended veto it is the misfire of. corpus_cjk.jsonl is regenerated because it is derived from the case table (test_cjk_corpus_matches_the_case_table pins it). Case classifications are UNDETERMINED pending the differential run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured, not guessed. The differential harness reports 739 names, 94 intentional diffs, 0 unexplained, exit 0 -- and every Latin row is byte-identical to the pre-#320 run, as the design predicted: _INITIAL is untouched and no Latin token carries a CJK character. The only three rows the run gained are the three names Task 3 added to the CJK corpus, each already claimed by an existing expected_changes.toml rule (comma-family, suffix-routing, cjk-comma-compound), so the toml needed no edit. All three rows are fix(#320). What 1.4.0 returns, via the pinned worker: '김민준, 씨.' -> first '씨.' / last 김민준 '김민준 씨.' -> first 김민준 / last '씨.' '田中さん, 様.' -> first '様.' / last 田中さん For the two comma rows that is exactly what 2.0 produced with Tasks 1-3 reverted, so both sat at parity with 1.4.0 until this change; the veto is what moves them. The spaceless-comma row 김민준 씨. is the one judgement call: it already differed from 1.4.0 beforehand, as given 김 / middle 민준 / family '씨.', because hangul segmentation divides the name either way. But it did not differ as the fields the row now asserts -- the suffix and the family-first reading both arrive with the veto's removal -- so it is classified to #320 rather than to the segmenter, matching how ko_honorific_ssi is classified to #307 without naming the same segmentation it also depends on. Each row's notes now record what 1.4.0 did with the input. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The initial veto stopped being a claim about shape alone, and several places had been written as though it still was. AGENTS.md's suffix_not_acronyms/is_an_initial bullet is now marked Latin-only. It enumerated a collision that no longer happens for the seven single-character CJK members of suffix_not_acronyms -- 씨, 様, 氏, 군, 양, 님 and 殿. The last of those was missing from the working list this pass started from; enumerating six of seven in a bullet that exists to name a set is the failure being fixed, so it was checked against SUFFIX_NOT_ACRONYMS rather than transcribed. _NO_INITIALS is cited as what it is -- the four scripts that do NOT have initials -- rather than glossed backwards. The two "kept in sync by hand" comments now say what stays in sync. _vocab's records that "verbatim" is a promise about the PATTERN, not the predicate: since #320, _INITIAL.fullmatch(text) and is_initial(text) are different questions and '씨.' answers them differently, so the bare pattern is not the thing to ask. The copy stays as verbatim as it ever was -- config's REGEXES["initial"] still differs by the empty alternative's `?`, exactly as the line above it and test_regex_sync already said, so this is not and never was a byte-identical copy. _render's records that its copy is deliberately NOT composed with the repertoire test -- layering forbids the import, and its only call site is v1's conjunction carve-out, which this pattern reaches only once `normalized in lex.conjunctions` holds. Verified rather than assumed: the default lexicon carries no CJK conjunction or particle, and neither does JA, ZH, RU or TR_AZ. That is a property of shipped data and not an invariant, conjunctions being configurable public API, so the comment also records why the divergence stays harmless if a user breaks it: CJK is caseless, so word.lower() and word.capitalize() agree. _classify's contract line said "Policy is not consulted", which since #320 is true of the Policy dataclass but not of the _policy module -- is_initial reads its _NO_INITIALS constant. Narrowed to Policy FIELDS, which is the point the line was making. docs/modules.rst's public `initial` tag gains the repertoire qualifier. It described an initial-shaped word; the tag now also requires a script that has initials. The release-log entry says what an upgrader sees. Every example in it was run -- the seven honorifics, the "田中さん, 様." peel and its periodless twin, and the three alphabets that keep their initials (А. С. Пушкин, م. الفارسي, Ա. Խաչատրյան) -- and the field claims are per-spelling, the vetoed honorific landing in the family name in "김민준 씨." and the given name only after a comma. It quotes v1's regex as REGEXES["initial"] actually spells it, `?` included, since that is the public name a reader will look up. It also records the finding from the differential: measured against 1.4.0, "김민준, 씨." and "田中さん, 様." were returning exactly what 1.x returns, so #307/#308/#312 shipped with a hole wherever the honorific carried a period. The differential README gains the trap this branch fell into twice. compare.py invokes the worker by script path; inserting `python` before it makes python the command and the script an argument, so uv never reads the PEP 723 metadata and the 1.4 pin is never installed. With nothing to satisfy, uv runs it in the project .venv, where the working tree is installed editable -- 2.x answering every query under a 1.4.0 label. The README says so and says what does NOT explain it, because the wrong story points at a mitigation that fails: sys.path[0] is the script's own directory, which holds no nameparser, so this is not the CWD trap AGENTS.md documents three sections up and PYTHONSAFEPATH=1 does not rescue it (measured: safe path on, still 2.0.0). Only an absolute script path from outside the project fails loudly. It is worse than an ordinary mistake because the corrupted output is the 2.x expected values: every diff vanishes, the run comes out as parity, and the reader draws the opposite of the truth. The fix is to stop trusting a version number nobody made the worker report. Plus the zsh note: `compare.py | tail` leaves $? as tail's status and PIPESTATUS undefined (zsh's is the 1-indexed pipestatus), so a failing run reads as passing. Prose only; no behavior change. Suite 2894 passed / 16 skipped / 11 xfailed, ruff clean, mypy clean over 95 files, sphinx-build -W clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #321 +/- ##
=======================================
Coverage 98.46% 98.47%
=======================================
Files 41 41
Lines 2807 2811 +4
=======================================
+ Hits 2764 2768 +4
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Two findings from the branch review, both about the same seam between "initial-shaped" and "is an initial". assign's roman-numeral fork asks whether the piece before a trailing single roman letter was an initial -- i.e. whether we are mid-run and the letter is a name part rather than roman five. It asked through the "initial" TAG, which #320 narrowed from initial-SHAPED to actually an initial. The fork only ever needed the shape, and the narrowing cost 'John 씨. V' its family name outright (given John, family ''), while 'John 김. V' turned a middle initial and a family name into a family name and a suffix. Blast radius was exactly that: last piece a bare roman numeral, preceding piece one CJK character plus a period. _vocab now names the two questions separately -- is_initial_shaped is v1's is_an_initial verbatim, is_initial is that ANDed with the repertoire test -- and assign calls the shape half on the token it used to read tags from. One shared definition rather than a hand copy: the repo has precedent for copied patterns (_assign._PERIOD_ABBREV) but each one buys a test_regex_sync obligation. All five probed inputs ('John 씨. V', 'John 김. V', 'John 田. X', 'John 김. V, PhD', 'John 김. 김. V') are back to their 1.4.0/master fields, and the #320 honorific fixes are untouched. The narrowed tag is NOT inert after this, which the review expected it to be. Measured over all three differential corpora plus every case- table row, with _classify's tag restored to the shape test: six field diffs, all of them #320's own fixes. The route is _group's _is_suffix_piece ('vocab:suffix' in tags and 'initial' not in tags) -- but as _assign imports it, in the trailing-suffix peel. Both call sites INSIDE _group (_is_rootname, and _is_suffix_piece reached through it) move nothing. Second finding: test_every_script_is_classified_for_initials compared the table to Script and its False rows to _NO_INITIALS, and never to is_initial. A Script.THAI declared with a private-use range, put in _NO_INITIALS with a False row, passed the whole test while is_initial('ก.') still returned True -- the declaration contradicted and nothing noticed. It also made the test the one on this branch on the wrong side of the no-constant-content rule. It now derives a representative character from that script's own _SCRIPT_RANGES entry and asserts is_initial on it, matching the row. Shape-admitted rather than simply the first codepoint, because several ranges open on an unassigned or punctuation character (HIRAGANA's 0x3040, KATAKANA's 0x30A0) that \w rejects for the shape's reason, which would make the repertoire assertion vacuous; a script with no initial-shaped character anywhere in its spans fails outright. Reproduced: under the scenario above the test now fails with "no character in _SCRIPT_RANGES[thai] is initial-SHAPED". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fix moves 24 of 52 probed input shapes and the branch pinned three. Four more rows, all measured before they were written, and all classified against 1.4.0 measured through the pinned worker (which reported 1.4.0, per the README's warning about the invocation that silently answers 2.x). - ja_honorific_with_a_period_no_comma, '田中さん 様.'. The SPACED form is the example is_initial's own docstring cites, and it was nowhere in the suite -- the existing row pins the COMMA form, which reaches the peel by flattening two runs where this one has a single run, so that row can stay green through a change that breaks this one. 1.4.0: first 田中さん, last '様.' -- what 2.0 gave until #320. - ko_honorific_period_under_strict_comma_suffixes, '김민준, 씨.' under Policy(lenient_comma_suffixes=False). The table had no lenient_comma_suffixes row at all, and this is the knob's own shape: a post-comma suffix that is initial-SHAPED. '씨.' was in that class by shape and the knob decided it; it is out of the class now and the honorific parses identically under both settings, which is what the row holds -- _script_segment names this setting as the one that strands 씨 in a neighbouring shape, and the knob's documentation scopes it to 'I' and 'V'. The knob has no v1 spelling, so the facade runner skips the row and the classification compares against 1.4.0's single reading (first '씨.', last 김민준). - ko_honorific_yang_trails / ko_honorific_yang_written_with_a_period, '김민준 양' and '김민준 양.'. 양 is what suffixes.py singles out as the shipped vocabulary's risk class -- a top-tier surname admitted on the strength of trailing position alone -- and only its LEADING reading was pinned. Both spellings now agree, so a future tightening of the 양/군 policy cannot move one without the other. 군 gets no pair: it parses identically and carries no surname reading, so it would pin nothing these two do not. corpus_cjk.jsonl regenerates from the case table (test_regex_sync pins it). The differential run stays at unexplained: 0 -- the three new CJK names classify against existing rules, 田中さん 様. and 김민준 양. under fix(suffix-routing) and 김민준 양 under fix(cjk-honorific-suffix); no rule was added or widened. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six prose corrections from the review pass; no behavior change. _types.py's STABLE_TAGS comment is the hand-maintained twin of docs/modules.rst's block, which the branch updated and this one did not -- it still promised the "initial" tag for any initial-SHAPED word, which since #320 is false for CJK. Nothing pins the two copies against each other, so the comment now says so. is_initial's docstring named the wrong mechanism: measured, '씨.' carried vocab:suffix and vocab:suffix-word before the fix as after, and is_suffix_lenient took it either way -- the veto rejected is_suffix_strict alone, via _is_suffix_strict_n. cases.py already had this right, so the docstring was contradicting the row it explains. _NO_INITIALS' rationale argued from two things that do not hold. It justified membership phonologically ("a syllable, not a letter"), but Devanagari is an abugida and Arabic an abjad and the branch's own tests assert both keep their initials -- so the stated criterion would have put Devanagari IN. The operative question is orthographic: does the tradition abbreviate a given name to one character plus a period? That matters concretely for Thai (#317), an abugida the old wording would misfile. And the enum does NOT admit only scripts that determine a convention -- KATAKANA is in it so the classifier can name what it declines, and neither DEFAULT_SCRIPT_ORDERS nor segment_scripts' default mentions it. The conclusion (do not derive _NO_INITIALS from the enum) survives on the better ground that membership is granted on assorted grounds. The release note claimed the fix for "a period", true only of U+002E: _normalize strips the ASCII period alone, so 씨． (U+FF0E), 씨。 (U+3002) and 씨｡ (U+FF61) miss the vocabulary lookup before the veto is reached, and all three parse identically before and after this branch. Scoped, with the still-open _normalize gap named. Also: test_classify's fixture comment and its docstring made opposite claims about whether й shipping is load-bearing (the docstring was right -- _LEX is local); that docstring named four scripts where the sibling test asserts six; cases.py said "fields" where only the field ASSIGNMENT matches; test_vocab called a digit "the" edge of \w when _ is equally inside it; and AGENTS.md's suffix_not_acronyms bullet now says which of the two predicates the tension is about. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
Author
Review pass — three commits addedFour reviewers went over the whole branch (code, tests, comments, silent failures). Mutation testing: 13 mutants, zero survivors, including the
|
This was referenced Aug 2, 2026
Owner
Author
The 'John 씨. V' incident was written out three times: is_initial_shaped's docstring, assign's call site, and a3c31f6's message. All accurate, and collectively noise. Keep the copy at the call site -- that is where someone debugging the input actually lands -- and leave the commit message as the full record. The docstring's job is helping a caller pick between the two predicates, so it keeps the choosing rule and the discriminating example and points at the fork rather than retelling it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #320. Blocks #319, which should go next on the corrected classification this establishes.
The bug
_vocab.is_initialused v1's^(\w\.|[A-Z])$. Python's\wis Unicode-aware, so씨.matched.is_suffix_strictappliesis_initialas a veto (V.inJohn V. Smithis a middle initial, not roman five) — so a veto written for Latin was being asked of scripts it was never about, and period-written CJK honorifics failed the strict suffix test.The cost ran past the honorific itself. Measured against 1.4.0:
김민준, 씨.씨./ last김민준김민준, suffix씨.田中さん, 様.様./ last田中さん田中, suffixさん, 様.Those two rows were at parity with 1.4.0 despite #307, #308 and #312 all being shipped. Those issues intended this behavior and did not deliver it — the veto stood in the way and 2.x quietly behaved like 1.x. Seven honorifics are affected: 様 殿 氏 군 님 씨 양.
The mechanism is the peel's site, not the comma structure:
_is_post_nominalasksis_suffix_strict, which returnedFalsefor様., so the scan-back stopped at it, took it as the site, found no listed tail there, and abandoned.StructureisFAMILY_COMMAbefore and after.The fix
Split the shape test from the repertoire test rather than narrowing the character class.
_INITIALstays v1-verbatim, so all three pinned copies — including publicconfig.REGEXES["initial"]— are byte-unchanged andtest_regex_sync's three-way relationship holds._policy._NO_INITIALSnames the four scripts whose characters cannot be an initial;is_initial_shapedis the shape half alone andis_initialANDs the two.Both halves have callers, and telling them apart is the point. The veto and the
initialtag wantis_initial._assign's roman-numeral fork wantsis_initial_shaped— it asks whether the preceding piece looks like part of an initial run, a question about layout. An earlier revision of this PR had it read the narrowed tag, which costJohn 씨. Vits family name outright; see the review section below.Narrowing to
[A-Za-z], which the issue originally proposed, would have been wrong in the other direction: it flipsЙ.from taginitialto tagconjunction— the regression the 2.1.0 Ukrainian entry claims to prevent — and strips the publicinitialtag from every Cyrillic, Greek, Arabic and Hebrew initial. Neither moves field output on a short name, so both pass a green suite.tests/v2/pipeline/test_classify.pynow asserts the tag._NO_INITIALSis enumerated rather than derived from_SCRIPT_RANGES. The criterion is orthographic convention — does the writing tradition abbreviate a given name to one character plus a period — not what a character is; stating it phonologically would misfile Devanagari, an abugida whose initials are real. And enum membership settles nothing about abbreviation:Script.KATAKANAis in it so the classifier can name what it deliberately declines, and keys no default behavior at all. A gate fails until any new member is classified, verified with a temporaryScript.THAI.Verification
-Wcleanunexplained: 0,expected_changes.tomluntouchedcases.pyrows, each classified from measured 1.4.0 values rather than predictedReview pass
Four reviewers went over the whole branch after the first five commits;
a3c31f6,08e61f6,2e9bee4and82f9815are the result. Two defects that per-commit review could not see:John 씨. Vlost its family name — the roman-numeral fork described above. Blast radius measured at 336 of 8,960 generated names, all one shape.is_initial('ก.')still returnedTrue. It now derives a character from_SCRIPT_RANGESand asserts behavior.Also four more pins (including the spaced
田中さん 様., which a docstring cited as the motivating failure while no test asserted it, and thelenient_comma_suffixes=Falsepath, which had no case row at all), a seventh doc place the six-place sweep missed (_types.py's hand-maintained twin ofdocs/modules.rst), and three corrected prose claims.Also here
tools/differential/README.mddocuments a trap this branch hit: insertingpythoninto the worker command bypasses its PEP 723 pin, so 2.x answers while labelled 1.4.0. The corrupted output is the expected values, so it reads as confirmation and would yield aparityclassification — the opposite of the truth.PYTHONSAFEPATH=1does not rescue it; the route is the project.venv's editable install.Scope
The release note is scoped to the ASCII period. Fullwidth and ideographic full stops, halfwidth katakana and NFD hangul miss the vocabulary by different mechanisms and stay open as #322. Also filed from this review: #323 (
양. 지훈splits the given name) and #324 (case-table bookkeeping).🤖 Generated with Claude Code