Repository navigation
Conversation
Collaborator
Author
|
Only one project in production could end up with data issues after this PR is merged, because it did the equivalent of having both EDIT: I believe that project can be taken off of hold status now, because the new logic won't cause those data issues. But I'll verify, then remove this comment if it's no longer accurate. |
Collaborator
Author
|
Currently having Claude do code review. Will post results when completed. |
LF stores a writing system tag as it was typed, but LCM canonicalizes a tag when it creates the writing system and thereafter knows it only by that canonical id. TryGet and GetWsFromStr both match the id literally (they are case-insensitive, but not structure-aware), so a tag whose canonical form differs structurally is never found: qaa-x-qaa-v -> qaa-x-v (redundant private-use "qaa" absorbed) th-Thai -> th (suppress-script dropped) LfWsToLcmWs then takes its else branch and calls Create() -- which DOES canonicalize -- so Set() collides with the writing system already there: Unable to set writing system 'qaa-x-kal' because this id already exists. The second test covers the quieter half: LfMultiText resolves each multitext key with GetWsFromStr and skips anything returning 0, so a value keyed by a non-canonical tag is dropped, and the clearing pass then blanks whatever LCM had for that field. testlangproj already contains qaa-x-kal and qaa-Zxxx-x-kal-audio, so these use a non-canonical spelling of an existing writing system rather than adding to the shared fixture project. The qaa-Zxxx-x-kal-AUDIO case already passes (case-only difference) and is a guard against regressing it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LF stores a writing system tag exactly as it was typed; LCM canonicalizes it when the writing system is created and thereafter knows it only by that canonical id. GetWsFromStr matches that id literally -- case-insensitively, but not structure-aware -- so a multitext keyed by a tag whose canonical form differs structurally resolves to 0: qaa-x-qaa-v -> qaa-x-v (redundant private-use "qaa" absorbed) th-Thai -> th (suppress-script dropped) WriteToLcmMultiString skips a key that resolves to 0, and its clearing pass then blanks whatever LCM held for that writing system, so the value is not merely dropped on the way in, it is deleted on the way out. Canonicalize the key before the lookup, here and in WsIdAndFirstNonEmptyString. LanguageTags.Canonical guards IetfLanguageTag.Canonicalize with IsValid, which it needs: Canonicalize throws on a tag it cannot parse, and a corpus scan found 48 such strings (custom field names that look like multitext keys). Verified against SIL.WritingSystems: Canonicalize agrees with the id liblcm itself stores for all 1251 valid tags in that corpus, and is idempotent on all of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TryGet matched the writing system id literally while Create canonicalized it, so a tag whose canonical form differs -- "qaa-x-qaa-v" becomes "qaa-x-v", because the private-use section repeats the "qaa" language subtag -- was not found, was created under its canonical id, and collided with the writing system already there: Unable to set writing system 'qaa-x-v' because this id already exists. GetOrSet does the lookup and the creation in one step and canonicalizes for both, so the two halves can no longer disagree. It returns true when it found an existing writing system and false when it created one, and a created one is already in the manager, so the separate Set call goes away and the two branches collapse into one path plus a conditional vernacular/analysis assignment. This implements only half of the TODO it replaces. The other half -- "use the ILgWritingSystemFactory interface and remove one point of FW 8-to-9 API incompatibility" -- does not follow: GetOrSet is declared on WritingSystemManager, not on ILgWritingSystemFactory, which offers only get_Engine, GetWsFromStr, GetStrFromWs, GetIcuLocaleFromWs and GetWritingSystems. The #if FW8_COMPAT blocks therefore stay, and the comment now records why rather than leaving a TODO that cannot be done as written. The vernacular comparison keeps comparing the tag as LF spells it against languageCode as LF spells it. Both are LF's own strings, so a project whose languageCode is itself non-canonical still matches its own input system. Braces added to the single-statement Abbreviation guard, which reads as indentation-scoped without them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LF has no vernacular/analysis flag on a writing system; the distinction is conventional, encoded in which config fields use it. LfWsToLcmWs treated exactly one writing system as vernacular -- whichever equalled the project's languageCode -- and filed every other one as analysis. A project whose lexeme field uses more than one writing system therefore lost the distinction: apltw2016's lexeme field uses qaa-x-IPA-dupl1 alongside th, and the phonetic one was made analysis. Take the vernacular writing systems from the fields that are vernacular by convention instead: citationForm, lexeme, etymology, senses.fields.examples.fields.sentence That list was already in MongoConnection, which applies the same convention in the other direction when it writes input systems back out to the config. Rather than have a second copy to keep in step by hand, it moves to MagicStrings.LfVernacularConfigFieldPaths and both directions read it from there. MongoConnection keeps its field name, so its six use sites are unchanged; the type widens to IReadOnlyList so the shared instance is not handed out as mutable. The example sentence path is nested, so the paths are walked a segment at a time. They are spelled as Mongo sees them, where each level of nesting goes through a "fields" document; in the mapped classes that is the Fields dictionary being indexed, so those segments are skipped. Where the config names no vernacular input systems at all, the project's language code is still used, so a project with a config LfMerge cannot read behaves as it did before. Tags are compared case-insensitively, matching LCM's own writing system lookups. This replaces the "needs to be a bit fuzzier" TODO in that method. Note the assignment still only happens for a writing system LfMerge has just created; one that already exists in the FieldWorks project keeps whatever vernacular/analysis membership it has there. That is unchanged behaviour, and it is why the new tests cover the derivation directly: creating a writing system in the shared test project cannot be undone afterwards, since WritingSystemManager has no removal API. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Some older projects had a missing or null DirtySR field, which was causing Mongo to throw an error because we never told it to treat null values as 0 for that int field. It's a simple one-line fix to treat missing values as 0, but treating null values as 0 requires a custom serialization helper. Still simple enough, thankfully.
The previous commit took the vernacular writing systems from a fixed list of config fields -- citationForm, lexeme, etymology and the example sentence -- shared with MongoConnection. Checking that against the 2026-07-06 corpus showed one list cannot be right for every project: 741 projects configure etymology with analysis writing systems and only 378 with vernacular ones, so naming etymology vernacular adds "en" to the vernacular list of hundreds of projects that have no English headwords. Classify per project from its own config instead. The lexeme and citation form fields are vernacular by definition and the sense definition and gloss are analysis by definition, so they anchor the decision; every other field carrying input systems -- etymology, the example sentence, custom fields -- is assigned by which anchor its writing systems already appear in. Only a writing system exclusive to one anchor counts as evidence: "en" is in both in many projects and so says nothing about any other field's role. This keeps what the fixed list already got right over the original languageCode rule. flh-flex has languageCode "flh-x-ortho", a writing system only its etymology field uses, so its real lexeme writing system "flh-x-cm" was classified as analysis; khg-yl-flex has the bare tag "khg" while its lexeme uses "khg-CN-x-Yangla" and "khg-fonipa"; odo has languageCode "th", which is not among its writing systems at all, leaving its 24,460 "en" headwords as analysis and the project with no vernacular writing system whatsoever. All three still come out right. Writing systems that overlap neither anchor are treated as vernacular and reported. 11 projects need this; grc-vie-flex is the clearest, a Koine Greek dictionary glossed in English and Vietnamese whose etymologies are in Hebrew and Aramaic -- neither vernacular nor analysis. The FieldWorks fields those feed are vernacular-typed, so calling them analysis would leave the data unreachable, whereas a spare vernacular writing system is inert. Since the two directions no longer share a rule, MongoConnection gets its own list back and MagicStrings.LfVernacularConfigFieldPaths goes away. MongoConnection answers a different question -- which config fields to write the vernacular writing systems into -- and that divergence is now deliberate rather than drift. languageCode survives as a last-resort fallback, for a config LfMerge cannot read at all: without it such a project would end up with no vernacular writing system, which is worse than the rule this replaces. Note this only affects writing systems LfWsToLcmWs creates. A project already synced under an earlier rule keeps the membership its FieldWorks project has. The tests move with the logic, keeping their coverage of the union, case-insensitive matching, the nested example sentence path, apltw2016's second lexeme writing system and the fallbacks. Two of them had described configs that do not match how Mongo spells them, with definition and gloss at the entry level rather than under senses; the anchor paths are strict about that, so those configs are corrected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SetCustomFieldData had no case for CellarPropertyType.MultiUnicode, so every multilingual text custom field fell through to `default: return false` and its data was dropped. The other direction has handled MultiUnicode all along (ConvertLcmToMongoCustomField line 363), so LfMerge could read one of these fields out of FieldWorks but never write one back in. That asymmetry is invisible in a log. The name lookup succeeds -- the field exists in LCM, so it is not among the remainingFieldNames that produce "Custom field X from LF skipped" -- and the default case returned false without saying anything, carrying a TODO asking for exactly the warning it never got. Migrating thai_jay is what turned it up: its "Top 10 list" field arrived in FieldWorks present but empty for all 537 entries, with nothing in 10,801 lines of LfMerge output to explain why. The comment above the switch is the likely origin, since it lists the types to implement and leaves MultiUnicode out; it is wrong, and is corrected here. This matters more than a missing type usually would, because MultiUnicode is the type every text custom field in a Language Forge project has. LF has never offered a single-string custom field, so nothing LF owns arrives as CellarPropertyType.String -- the one text type that did work. Alternatives LF does not have are cleared rather than left in place. That is safe here specifically because the export writes every alternative LCM holds, via LfMultiText.FromMultiITsString: an alternative missing from LF is one a user deleted there, not one LF never saw. It also matches what LfMultiText.WriteToLcmMultiString does for the built-in multitext fields. Values that have not changed are left untouched, as the String case does, so they stay out of the .fwdata XML and out of the Mercurial commit. MultiString is deliberately not added. GetCustomFieldData cannot produce one either, so an import path for it would be untested code for a shape LfMerge never emits; it now reaches the new warning instead of disappearing. The default case logs a warning, which is what the TODO asked for and what the LCM-to-Mongo direction has always done. A field type going unimplemented is worth a line in the log rather than silence. Tests cover the four cases against the test project's Cust_Single_Line_All, which really is MultiUnicode with English and French alternatives: both alternatives written, a previously emptied field written (the migration case), an alternative LF dropped cleared rather than left stale, and an unidentified writing system skipped. All four fail on the unmodified converter, each showing the field still holding its original "Some custom text". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude determined that this warning is no longer needed; there are indeed some LF fields (usually on the sense level) that have empty but non-null multitexts, but they truly are empty and there is no loss of data involved, so no need for a spammy log warning.
The MultiUnicode case added to SetCustomFieldData looked each LF key up with GetWsFromStr as LF spells it. LCM knows a writing system only by its canonical id, so a non-canonical key such as "fr-Latn" (canonically "fr") resolved to 0 and was skipped as unidentified. The clearing pass then blanked that writing system's alternative, since nothing from LF had claimed it: the LF value was not merely dropped, the LCM value was deleted. This is the same bug 0c70604 fixed in LfMultiText.WriteToLcmMultiString for the built-in multitext fields; canonicalize the key here the same way. The new test fails without the fix, French having been cleared. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ClassifyVernacularWritingSystems returned only a vernacular set, and LfWsToLcmWs filed every new writing system in exactly one list: vernacular if the set held it, analysis otherwise. That got three shapes wrong. A field with a vernacular-only writing system had ALL its writing systems made vernacular, including ones exclusive to the analysis anchors: an etymology configured as [seh, pt] made Portuguese vernacular-only, so a new pt writing system was never added as analysis and pt glosses had nowhere to go. The field's evidence now decides the role of the field, not of every writing system in it; an analysis-only writing system stays analysis. A writing system in both anchors -- "en" in lexeme and gloss alike -- was vernacular, and so never analysis. The classification now returns an analysis set as well, and a new writing system is added to every list it belongs in. A writing system in a field overlapping neither anchor was made vernacular only, on the reasoning that a spare vernacular writing system is inert. It is not: a French-only notes custom field left French out of the analysis list, so an analysis-typed FieldWorks field would not offer it. With no evidence either way such a writing system is now both, which serves vernacular-typed fields (the etymology form, the example sentence) and analysis-typed ones alike. The integration test fails without the LfWsToLcmWs change, the new writing system missing from the current analysis list. This commit also removes the BOM from ConvertMongoToLcmLexicon.cs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
0c70604 and c91a6c7 canonicalized LF's tags at three GetWsFromStr call sites, but the other places that resolve a tag LF supplied still matched it literally, so each had the non-canonical-tag bug those commits fixed: - BestStringAndWsFromMultiText compared LF's keys with ws.Id. A key such as "fr-Latn" never matched French, so a single-string lexicon field took the text of whichever writing system came next in the search order, or the first key in the document, instead of the preferred one. - The String custom field case resolved the key it picked literally, and gave up on the whole field when that returned 0, so the LF edit never reached LCM. - A span's lang attribute resolved to 0, and building the TsString then threw ArgumentOutOfRangeException ("The specified writing system code is invalid"). - A multi-paragraph's input system and the option list item writing system were resolved literally too. LanguageTags.WsIdFromLfTag is now the one way to resolve an LF tag: the canonical tag first, then the tag as LF spells it, for a writing system LCM might hold under a non-canonical id. Keys are still resolved at lookup rather than rewritten when the multitext is read in, since LF shows text under the tags its own config spells, and rewritten keys would not be shown. LfMultiText gains BestStringAndWsId, which resolves each key to a handle before comparing it with the search order, and whose fallback skips values in writing systems LCM does not know. BestStringAndWsFromMultiText and the String custom field case both use it. The MultiUnicode custom field case re-implemented WriteToLcmMultiString, and the two copies had already drifted: that is how the copy came to lack the canonical lookup that c91a6c7 added. Both now call LfMultiText.WriteToLcm, which reaches LCM through delegates, since a custom field has only ISilDataAccess and not an IMultiAccessorBase. The built-in fields gain two behaviours the custom field copy had: an unchanged alternative is left alone, keeping it out of the .fwdata XML and the Mercurial commit, and a null value is treated as empty rather than throwing. The String custom field and span tests fail without the fix; the others test the new methods directly. All 219 LCM tests pass. The 21 Mercurial and synchronize tests that fail do so because hg init and hg serve cannot run in this environment. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ClassifyVernacularWritingSystems decides whether a writing system is vernacular
or analysis from the project config alone, and the config is weaker evidence
than it looks: Language Forge routinely offers every input system in every
field, so a tag configured for the lexeme field may have no headword in it
anywhere. Scanning the 2026-07-06 corpus, 750 (project, writing system) pairs
are configured somewhere but hold no text at all.
Counting the text says what a writing system is really being used for. This
adds the counting pass on its own; nothing consumes it yet.
Fields are keyed by the dotted path the config spells them with, nesting through
"fields" ("senses.fields.examples.fields.sentence"), so the counts line up with
the paths classification already works in. Every built-in multitext field is
counted, not just the ones classification consults today.
Deleted entries are skipped: they are tombstones Language Forge keeps so the
deletion can be replayed, and their text is not evidence that anyone is using
that writing system. Whitespace counts as empty. Tags are matched
case-insensitively, as every LCM writing system lookup is, because the config
and the lexicon do not always agree on the case of a tag.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The config says which writing systems a field OFFERS, and Language Forge routinely offers every input system in every field. ClassifyVernacularWriting- Systems believed it, so a project whose lexeme field lists "en" alongside the vernacular got "en" classified vernacular even when not one headword is written in it. That is the shape behind most of the corpus's unresolved writing systems: 168 of the 323 have lexeme and gloss offering exactly the same input systems, so neither anchor has any exclusive tag and every other field becomes unresolved. Where the lexicon has text to show for a writing system, that text now decides, and the config only answers for writing systems with nothing written in them. By share, not by presence. A few strays are not evidence of a second role: in brb-flex-2022 the phonetic writing system has 4,030 headwords and one stray string in an example reference, and FieldWorks has it as vernacular only, while in cmo-khm-flex the vernacular has 7,815 vernacular strings against 300 in glosses and is likewise vernacular only. The one writing system those projects really do list as both sits at 28%, so 10% is a round number in the middle of the gap. What no text can settle, language often can. A writing system nothing else has placed still has a language, and the other writing systems spelling that language have usually been placed already: a phonetic or audio spelling of the vernacular is vernacular too, and a regional spelling of the analysis language is analysis. The language of a tag, for this purpose, is its language subtag plus any private-use subtags that name the language rather than a variant of it. Script and region come out, so "en" and "en-GB" are one language. The private-use subtags have to stay, because under "qaa" -- the code for a language with no code of its own, which Language Forge projects lean on heavily -- they ARE the language: "qaa-x-kal" and "qaa-x-hbo" are no more the same language than "fr" and "de". The phonetic markers, the audio marker and the "dupl1" suffixes FieldWorks appends all come out, so "qaa-x-kal", "qaa-fonipa-x-kal" and "qaa-Zxxx-x-kal-audio" are one language. A language both roles already use settles nothing, for the same reason a writing system in both anchors is no evidence of a field's role. Checked against the four FieldWorks repositories on hand, whose LanguageProject.langproj records the real vernacular and analysis lists: this gets 18 of 18 writing systems right, where the config-only rule gets 15 right and 2 wrong (cmo-khm-flex's km-KH and xkk-flex-2022's en, both wrongly vernacular). Across the 2026-07-06 corpus it takes the projects with at least one unresolved writing system from 323 to 69, and the unresolved (project, writing system) pairs from 419 to 71. A writing system the config forgot but the lexicon uses is now classified too, rather than falling through as neither. Counting means a second streaming pass over the Mongo lexicon before any writing system is created. GetLexicon yields from a cursor, so the lexicon does not all sit in memory at once. WithNoAnalysisAnchorEveryOtherFieldIsUnresolved covered a phonetic spelling of the lexeme's own language, which language affinity now settles, so it grows a writing system of an unrelated language to keep making its original point. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…vidence Only the lexeme, citation form, definition and gloss counted as fields whose role FieldWorks fixes. Four more do: an example sentence is vernacular-typed, and an example translation, an entry note and a literal meaning are analysis- typed. Counting the 2026-07-06 corpus for each field, the projects whose text there is exclusively one role against exclusively the other: example sentence 332 vernacular : 5 analysis example translation 1 vernacular : 320 analysis entry note 0 vernacular : 255 analysis literal meaning 4 vernacular : 136 analysis Taking them as evidence takes the projects with at least one unresolved writing system from 69 to 61, and still gets all 18 writing systems in the four FieldWorks repositories on hand right. The pronunciation field is deliberately left out although it looks vernacular at 114 projects to 16. That is 12% analysis, above the share a single writing system may hold in the other role and still count as playing one role alone, so projects evidently use the field for more than one thing and it cannot be trusted to settle anything. The field paths are now the constants LfWritingSystemUsage already defines, so the counting pass and the classification cannot drift apart on a spelling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The field loop takes its evidence as it stood before the loop began, so a field holding only writing systems the loop itself has settled looks like evidence of nothing, and everything in it goes back into doubt. A writing system settled by one custom field can therefore be dragged into doubt by another. A second pass settles those again from the company they keep: a custom field holding the vernacular and one writing system nothing has placed is being used for vernacular content, so the unplaced one is vernacular too. Only unanimous company counts, for the same reason a writing system in both anchors is no evidence of a field's role. This takes the projects with at least one unresolved writing system from 61 to 57, and still gets all 18 writing systems in the four FieldWorks repositories on hand right. The pass deliberately does not chain: the company is counted once, before any of it is settled, so a writing system whose only company was itself in doubt stays unresolved rather than inheriting a second-hand answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The field loop's no-evidence branch put every writing system in a field into doubt, including ones something had already placed. Before the companions pass that cost nothing but a log line, since a doubtful writing system goes in both lists anyway. With it, it strips a role. Take a config whose lexeme offers [seh, en], whose gloss and definition offer [en], and whose general note offers [en], with nothing written in en yet. The anchors correctly put en in both lists. The note holds nothing exclusive to one role, so the loop also put en in doubt; the companions pass then found that the only single-role company en keeps is seh, in the lexeme field, and removed en from the analysis list. A new "en" writing system was created vernacular only, and English glosses and definitions lost their analysis writing system. A field with no evidence now puts in doubt only the writing systems nothing else has placed. One already placed keeps its place. In the 2026-07-06 corpus five writing systems were losing a role this way. The fix also stops flagging the 47 empty projects whose "en" is offered in every field, both anchors included, so the projects with a writing system in doubt go from 57 to 10. It still gets all 18 writing systems in the four FieldWorks repositories on hand right. Two tests described the old behaviour. AWritingSystemInBothAnchorsIsNotEvidence expected en itself to be in doubt; it now gives the etymology a second, unplaced writing system to keep making its point. The companions tests relied on a writing system being dragged back into doubt, which no longer happens, so they now list the doubtful field first -- fields are visited in config order -- and the "company is itself unsettled" case uses two writing systems with no other company at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BestStringAndWsFromMultiText ended with a "last-ditch" fallback: take the first
non-empty value whatever writing system it is in. Once 74e1684 gave it a
fallback that already takes the first value in any writing system LCM has, the
last-ditch branch was reached only when no key resolved at all, and then it
always returned writing system 0. Building a string in writing system 0 throws
("The specified writing system code is invalid"), and the per-entry catch in
RunConversion then abandons the entry half-written: every field converted
before the throw is kept, and every field after it is skipped.
Returning null instead, as the review suggested, would be worse. Every caller
assigns the result straight into the LCM property, so null clears the field,
deleting what LCM holds on account of text that could not be placed.
So the field now keeps its current value, and the log says which writing
systems' text had nowhere to go. Null, clearing the field, still means what it
always meant: LF holds no text there. The callers pass the current value in.
The picture caption had the same hole: a caption in no writing system LCM has
came back as writing system 0 and threw in GetOrCreatePictureByGuid. It now
starts the picture with an empty caption in the default analysis writing
system, and SetMultiStringFrom fills in every alternative that can be placed.
WsIdAndFirstNonEmptyString, which fed both, is gone.
The new test fails without this: the entry is abandoned at the scientific name,
so the sociolinguistics note converted after it never arrives.
The etymology-source line changed here is compiled only for DBVERSION_7000068,
which does not build against the current SIL.LCModel for reasons unrelated to
this change (ILexEntry.EtymologyOA and ILexEtymology.Source are gone), so it is
unverified.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LCM knows a writing system by its Id, which is whatever its .ldml file is
named, and also has its LanguageTag, which is always canonical. In older
projects the two differ. brb-flex-2022 holds "km-Khmr-KH.ldml", whose identity
is km plus KH, so LCM holds it as Id "km-Khmr-KH" with LanguageTag "km-KH"; so
do spt-flex's "hi-Deva-IN" (hi-IN) and "spt-Tibt-x-tib" (spt-x-tib). LfMerge
writes LF's multitext keys by Id and the project's input systems by
LanguageTag, so LF can hand back either spelling.
WsIdFromLfTag tried the canonical form first, which deletes data in the
brb-flex-2022 layout. That project also holds a second, unused writing system
whose Id is "km-KH". Text keyed by the real writing system's Id, "km-Khmr-KH",
canonicalized to "km-KH" and went to the unused one, and WriteToLcm's clearing
pass would then blank the real writing system's text, as missing from LF, in
every entry LF sent back. (This branch has never run against that project, and
LF has never edited it, so nothing has been lost.)
It now tries, in turn:
1. The tag as LF spells it, as an Id -- the spelling LfMerge itself exports.
2. Its canonical form, as an Id.
3. Its canonical form as a LanguageTag, which finds spt-flex's "hi-Deva-IN"
for the "hi-IN" in LF's config, where before it found nothing and the
edit was dropped. Should several writing systems share the LanguageTag,
the one created first wins.
The two new tests reproduce those layouts and fail without this: the first
resolves to the unused writing system, the second to nothing.
Trying the exact Id first is never worse than develop, which only ever tried
it. One consequence is worth knowing. cmo-khm-flex's LF entries still carry a
few stale keys spelled "km-Khmr-kh", from before the project's Khmer writing
system was renamed "km-KH"; they resolve, as they do on develop, to the unused
"km-Khmr-KH" writing system LfMerge created in 2022 rather than to "km-KH".
Sending them to the writing system in the project's lists instead needs the
LanguageProject, which this function does not have, and needs a rule for which
of two keys resolving to one writing system wins, which does not exist yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LfWsToLcmWs looked each LF input system up with GetOrSet, which looks a tag up only as an Id, and created a writing system when that found nothing. LfMerge exports input systems by LanguageTag, while LCM knows a writing system by its Id, and in older projects the two differ: spt-flex's input system "hi-IN" is the writing system LCM holds as "hi-Deva-IN". On the next sync GetOrSet would not find it, would create a second writing system "hi-IN" and add it to the current lists. WsIdFromLfTag, which every multitext key goes through, then matches that duplicate exactly, so text typed into LF's hi-IN field goes to a writing system FieldWorks shows as new and empty rather than to hi-Deva-IN. The input system is now found the way every other LF tag is found, with WsIdFromLfTag, and a writing system is created only when that finds nothing. The new test fails without this: the duplicate appears among the project's writing systems. This was already the case on develop, which also used GetOrSet here. The LanguageTags comment claiming GetOrSet made the resolver unnecessary for LfWsToLcmWs was wrong, and is corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After applying an input system's abbreviation and direction, LfWsToLcmWs called wsManager.Replace(ws). ws is already the manager's own object, so the edits are in place without it, and Replace can only do nothing or do harm. It looks the writing system up by LanguageTag, which in older projects is not the Id LCM knows it by, and then: - If no writing system has that LanguageTag as its Id, Replace falls through to Set(ws), which gives the writing system a fresh handle mid-sync. That happens whenever an input system is spelled by an Id that is not its LanguageTag, as in projects LfMerge set up under FieldWorks 8, which exported input systems by Id: to judge from those exports, aeb-ra-flex holds "aeb-Arab" (LanguageTag "aeb") and nod-wlist-flex holds "th-Thai" (LanguageTag "th"). - If another writing system has it -- brb-flex-2022's unused "km-KH" beside the real "km-Khmr-KH" -- Replace removes that one from the manager and gives its handle to ws. The call is gone. The new test covers both layouts, and fails without this because the writing system's handle changes; the eviction in the second layout is from reading Replace, since the test stops at the handle. The test also checks that the input system's abbreviation is still applied. This was already the case on develop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LfMerge spelled a writing system two ways in Language Forge. Every multitext key comes from GetStrFromWs, which gives the writing system's Id. The project's input systems, its languageCode, the config's field lists at initial clone and the custom-field config came from its LanguageTag, which is always canonical, while the Id is whatever the .ldml file is named. In older projects the two differ: brb-flex-2022's Khmer is Id "km-Khmr-KH" with LanguageTag "km-KH", and che-flex's vernacular is Id "ce-la" with LanguageTag "ce-LA". LF's editor matches a field's input systems against an entry's keys exactly and case-sensitively, so it showed all of that text as empty: in the 2026-07-06 mongodump, twelve projects are affected, che-flex's 18,423 headwords among them. All of them are now spelled by GetStrFromWs too, through LfTagOf, so the input systems and the keys cannot disagree. FieldWorks 8 builds did the same, by Id; the three #if FW8_COMPAT blocks here existed only to choose between the two spellings, and FW8_COMPAT is never defined, so they are gone. Spelling by Id also agrees with LCM, which looks every writing system up by Id, so a key LF hands back resolves at the first step of WsIdFromLfTag. The project records already spelled by LanguageTag need re-spelling once, since LfMerge rewrites the input systems and custom-field config on every sync but built-in field config only at initial clone. That is done outside LfMerge, by a mongosh script. The new test fails without this: the input system is exported as "nl-BE" while the gloss is keyed "nl-Latn-BE". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A project's current vernacular, analysis and pronunciation lists are stored as strings of Ids. If one names an Id LCM has no writing system for, the export threw KeyNotFoundException before writing anything to Language Forge: LINQ's Select and ToList read a list through its indexer or CopyTo, and liblcm's versions of both throw for such an Id. Only its enumerator tolerates one, by yielding null. The three lists are now read with foreach, and an Id with no writing system is left out with a warning. This makes the pronunciation list safe, which nothing else in the export reads. A vernacular list in that state would still fail, inside liblcm's own GetWritingSystemList when the custom-field config is written; FLEx would hit the same thing, so such a project is unlikely to exist. The new test fails without this, with the KeyNotFoundException above. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LanguageOf looked for the private-use "x" from the second subtag on, so a tag that begins with it, such as "x-kal", came out as just "x". Every such tag then counted as one language, and language affinity could place "x-hbo" in the same role as "x-kal". The 2026-10-01 corpus has two tags like this, "x-IPA" and "x-v". The search now starts at the first subtag, so "x-kal" comes out "x-kal" and "x-kal-dupl1" still matches it. The new test fails without this: "x-hbo" is classified with "x-kal" instead of being left unresolved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A custom field whose type the LF to LCM direction does not implement, such as an owning collection or sequence, logged its warning for every entry, sense or example that had it, so a project with 10,000 senses logged it 10,000 times on every sync. That is the same kind of log spam d90c0b9 removed. The converter now remembers which fields it has reported and warns about each once. One converter is made per transfer, so that is once per field per sync. The new test fails without this: three entries give three warnings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A config field that is not an anchor is classified by the writing systems it
offers that belong to one role only. A field offering a vernacular-only and an
analysis-only writing system side by side took the vernacular branch, only
because that branch was checked first, and the other writing systems it offered
became vernacular. That is the opposite of the rule ResolveByCompanions follows,
where only unanimous company counts.
Such a field is now evidence of neither role. Writing systems it offers that
nothing else has placed are left unresolved, as for a field overlapping neither
role, and the companions and language-affinity passes then decide them where
they can.
In the 2026-10-01 corpus 166 projects have 1,381 fields like this, but only 4
projects and 5 writing systems change, measured with a simulation of this
classifier. Compared with the roles FLEx gives them (the inputSystems record
LfMerge exports):
- kam-flex: "es" and "fr", in a wordlist field, were vernacular and are now
both; FLEx has them as analysis.
- mni-flex: "zh" was vernacular and is now both, as FLEx has it.
- msea-thai-flex: "th-x-latin" was vernacular and is now both; FLEx has it as
vernacular.
- xin-flex: "xin", which LCM does not have yet, was vernacular and is now
analysis, through language affinity with its analysis writing system
"xin-GT-fonipa". It holds only pronunciation text, so vernacular was the
better answer there; the project's writing systems look mislabelled.
Only xin-flex is affected in practice: the others already exist in LCM, and
classification only decides the lists for a writing system LCM does not have.
The new test fails without this: "fr" is classified vernacular-only instead of
left unresolved.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FieldWorks draws its pronunciation writing systems from the vernacular ones: liblcm's InitializePronunciationWritingSystems considers no others. But the classifier ignored the pronunciation field's text altogether, since the field is not an anchor, so a writing system used only for pronunciations got whatever role its company or language suggested. xin-flex's "xin", which holds 226 pronunciations and nothing else, became analysis-only after 7b76f8f. A writing system with no text in the anchor fields but some in the pronunciation field is now vernacular. That only adds the role, it does not settle it, so the config can still add analysis: the field stays out of the anchors, because 12% of the projects writing in it write analysis there, and a stray pronunciation string should not take an analysis role away. Measured with a simulation of this classifier on the 2026-10-01 corpus, against the roles FLEx gives the 564 writing systems LfMerge's export records: 516 agree, up from 515, and none gets worse. Projects with an unresolved writing system go from 10 to 7. 18 writing systems in 18 projects change, 8 of them in Send/Receive projects, nearly all phonetic ones becoming vernacular. In dtlat-flex, Swedish, configured for the gloss with a single pronunciation string, becomes both rather than vernacular only. This is the part of the four-set design measured best. The rest of it -- "both" only for a genuinely split writing system, with a tie-break for one offered in both anchor groups with no text -- lost five roles FLEx gives, and adding to FLEx's pronunciation list is left out, since LfMerge only reads it. ThePronunciationFieldIsNotEvidenceOfAnything asserted that pronunciation text leaves a writing system unresolved; it is now ThePronunciationFieldIsNotAnAnchor, for a writing system merely offered there. The two new tests fail without this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every Mongo to LCM transfer read the whole lexicon twice: once to count the text in each field for classifying writing systems, once for the entries themselves. The classification is only used to put a writing system LCM does not have yet into its lists, which after a project's first sync almost never happens, so on nearly every sync the first pass was thrown away. For a project like spt-flex that is 18,600 entries read for nothing, every time. LfWsToLcmWs is now given a way to count the lexicon rather than the count, and classifies when the first writing system new to LCM turns up. The classification depends only on LF's config and text, so creating writing systems before then cannot change it. With no new writing system the lexicon is not counted, and the notice about writing systems whose role could not be determined, which was logged on every sync, is not logged either. The counting pass also counted about thirty fields when classification reads only nine: the vernacular and analysis anchors and the pronunciation field. It now counts just those, so classification gives the same answers. The test double for Mongo now counts lexicon reads. The new test that a transfer with no new writing system reads the lexicon once fails without this: it reads it twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The field loop placed writing systems field by field, in the order the config lists its fields, and a field with no evidence of its role put in doubt only the writing systems not placed yet. So a writing system offered in such a field and in one that does have a role was placed when the field with a role came first, but put in doubt when the field of no role came first. Once in doubt, the companions and language-affinity passes could strip one of the roles the other field had given it: en-fonipa, offered in a vernacular field and an analysis field, ended up analysis only if a field of no role came before them, and both otherwise. Every field's evidence is now gathered first, and a field of no role puts in doubt only the writing systems that neither the anchors nor any field with a role places, wherever the fields come in the config. Measured with a simulation of this classifier on the 2026-10-01 corpus, under the config's own order, its reverse and 20 random orders of every project's fields: no project's answer depended on the order before this, and none does now, and in the config's own order nothing changes. So this is a fix for a config the corpus does not yet hold, and changes no project's result today. The new test fails without this when the field of no role is listed first. AWritingSystemInDoubtIsSettledByTheCompanyItKeeps still passes; its comment described the old field-by-field placing and is updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9491fac's test checks the handles of a writing system whose Id is not its LanguageTag and of one whose Id is that LanguageTag, but nothing carried text in both through a sync: the xkk-flex-2022 layout, where re-registering the first evicted the second and took its handle. The new test gives each a gloss of its own and takes them through an export to LF, an LF edit of both, an import and a second export. Each must keep its handle and its own text in LCM and in LF. It uses ca-Latn-AD and ca-AD, whose LanguageTag is the same under both SLDR datasets; xkk-Khmr's is not, and the tests run on the older one. With the Replace call 9491fac removed put back, the test fails: the second export throws a NullReferenceException in LcmWsToLfWs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An LF multitext can hold two keys that resolve to the same LCM writing system. It happens when LF's config spells a writing system otherwise than LfMerge's export does: the export keys every value by the writing system's Id, an LF user edits under the config's spelling, and the export's value stays in the entry, hidden from LF's editor, until the entry is exported again -- which, once the edit is written to LCM and the dates match, may not be for a long time. In spt-flex, whose config says "hi-IN" for the Id "hi-Deva-IN", any entry edited in LF would hold both. WriteToLcm wrote the keys in document order, so the later one silently replaced the earlier. The keys are now resolved before anything is written, one is chosen for each writing system, and the other is reported: the built-in fields and the MultiUnicode custom fields both log a warning naming the two keys and the start of each one's text. The key chosen is the one spelled as the project's config spells the writing system, since that is what LF's editor shows and so the only one a user can have edited. Before the config is re-spelled by Id, that is the user's edit and not the hidden export; after, it is an edit made in place under the Id and not a key left over from before. Where both spellings or neither are in the config, it is the later key, as it always was: LF adds a new key after the existing ones. Preferring the key spelled as the Id would drop the user's edit in every project whose config has not been re-spelled. Preferring the value that differs from LCM would write the user's edit once, and then, on the entry's next sync, write the stale hidden value back over it. In the 2026-10-01 corpus only one multitext holds such a pair, a huz-flex lexeme with the same text under "huz-Cyrl" and "huz". (cmo-khm-flex's "km-Khmr-kh" and "km-KH" resolve to two writing systems, and xkk-flex-2022's "xkk" and "xkk-Khmr" are each a writing system's exact Id.) The new tests fail with either the Id spelling or the later key always preferred: three cases and two respectively. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LfWsToLcmWs gives a writing system new to LCM the roles the classification found for it, looking its input system's tag up in the classified sets. Those sets spell each writing system as the config and the lexicon do, which need not be how the input system spells it. An LF user who adds the input system "qaa-x-qaa-new" and configures the lexeme field with "qaa-x-new" got a writing system that was not found among the vernacular ones and so was made analysis. The lookup now matches any tag with the same canonical form. That cannot merge two writing systems: this one is new to LCM, and had LCM held any writing system with that canonical form, WsIdFromLfTag would have found it. This is half of what was proposed for this finding. The other half -- counting text under every spelling of a writing system together -- was measured with a simulation of this classifier on the 2026-10-01 corpus and left out: against the roles FLEx gives the 564 writing systems LfMerge's export records, it made agreement worse, 516 to 515. ket-flex's "ket" took the ten analysis-field strings of "ket-Cyrl", which FLEx has as vernacular, and became analysis only where the config alone had made it both. The lookup on its own changes no writing system there. The new test fails without this: the writing system is made analysis only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The field loop, ResolveByCompanions and ResolveByLanguageAffinity each worked out which writing systems are vernacular and not analysis, and analysis and not vernacular -- the first two with the same four lines, the third with two loops -- since each pass weighs evidence after the passes before it have changed the sets. ExclusiveRoles now does it for all three, so a change to how it is done, such as to the case-insensitivity, cannot reach one copy and miss the others. No behaviour changes; the tests are unchanged and pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SetMultiStringFrom, which writes every built-in multitext field, gave WriteToLcm no callback for a key that resolves to no LCM writing system, so the text under it was skipped without a word. It never reaches FieldWorks, and it leaves LF as well once FieldWorks next changes the entry and the export rewrites the field from LCM. The custom fields and the single-string fields already logged this case; the commonest path did not. It now logs a warning naming the key and the start of its text, so the entry can be found. WriteToLcmMultiString takes the callback, as WriteToLcm does. In the 2026-10-01 corpus this is rare: of the keys outside each project's inputSystems record, all but 6 were written by LfMerge's own export and so name writing systems LCM has, just outside its vernacular and analysis lists. The 6 hold 13 values, all in entries LF users edited; one, in spa-par-flex, would be written on its next sync. The new test fails without this: nothing is logged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ClassifyByText's comment said the rule that makes a writing system with only pronunciation text vernacular only added that role, so that the config could still add analysis, and that a stray pronunciation string should not take an analysis role away. That holds where the config offers the writing system in an anchor field. It does not where the config offers it only in custom fields: there, once placed, it is not put in doubt by a field of no role, and a custom field it alone is offered in is taken for a vernacular one, so it is vernacular only. A code review found the mismatch. The behaviour is kept and the comment now says what it is. In the 2026-10-01 corpus all 57 writing systems with pronunciation text and no anchor text are pronunciation writing systems -- phonetic, IPA and audio ones, and xin-flex's "xin" -- and FLEx has as vernacular each of the 11 whose role LfMerge's export records. The only stray strings, Swedish in dtlat-flex and English in two LF-only test projects, are in writing systems the config offers for an anchor field, so they are both. Applying the rule after the other passes instead, as the comment implied, was simulated: it agreed with FLEx no better, and turned 11 phonetic writing systems from vernacular only into both. The new test records that a writing system offered only in a custom field, with pronunciation text and nothing else, is vernacular only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
None of these change behaviour. - LanguageTags.WsIdFromLfTag, LfWsToLcmWs and a writing-system test said LfMerge exports input systems by LanguageTag; 2377f89 made it spell them by Id, so they now say it used to. - WsIdFromLfTag said that of several writing systems sharing a LanguageTag the one created first wins. It is the one with the lowest handle, and LCM numbers writing systems in the order it loads them. - ClassifyVernacularWritingSystems' summary named the old anchors, listed the example sentence among the fields assigned by overlap, and said nothing of the text, the pronunciation rule, the order-independent field loop, the companions or language affinity. It now describes all of them. Its returns comment cited grc-vie-flex and "11 projects"; in the 2026-10-01 corpus it is 7, and grc-vie-flex is no longer one of them, so it now cites kam-flex. - LfWsToLcmWs said roles are worked out from the config; they are worked out from the config and the text. - HasRole now says that a role found under one spelling counts for all of them, even where the text decided otherwise under another -- which can only add a role. - The doc comment for LcmWsToLfWs had been left above LfTagOf, giving it two summaries; it is back on LcmWsToLfWs. That method's FW 8 comment no longer claims the code stays compatible with FW 8. - LfTagsOf said it keeps a missing Id from ending the transfer. That holds for the pronunciation list; one in the vernacular or analysis lists still ends it, in LcmWsToLfWs. - TheExampleSentenceIsClassifiedLikeAnyOtherField is now TheNestedExampleAnchorsAreFound, the example sentence being an anchor. - A test comment said LfMultiText resolves keys with GetWsFromStr; it used to. - The writing-system tests' restore helpers said the mock project record outlives a test. Each test gets a fresh one; what outlives a test is a writing system added to LCM. - LexiconReads said each read is a pass over Mongo. Filtered requests count too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A code review found tests that would still pass with the code they cover broken: - AWritingSystemInDoubtIsSettledByTheCompanyItKeeps passed with ResolveByCompanions deleted. Since f0da10e its "abc" is placed by the field loop, and the writing system the companions pass does settle, "xyz", was never asserted. It now is. - LcmToMongo_PronunciationListNamingAWritingSystemLcmDoesNotHave_LeavesItOut only showed the export did not throw: its assertion was about the inputSystems record, which the pronunciation list never feeds. The Mongo double now keeps the vernacular, analysis and pronunciation lists it is given, and the test checks the pronunciation list holds the writing system LCM has and not the one it lacks. - LcmToMongo_WritingSystemWhoseIdIsNotItsLanguageTag_IsSpelledByIdEverywhere checked only the record and a gloss key. It now checks the analysis list and the custom-field config are spelled by Id too. Each now fails with its code broken: with ResolveByCompanions skipped, with LfTagsOf returning nothing, and with custom-field input systems spelled by LanguageTag. The double's LexiconReads also counted filtered requests, such as the comment converter's one per comment, as passes over the lexicon, so MongoToLcm_NoInputSystemNewToLcm_ReadsTheLexiconOnce held only while the sample data had no comments. Filtered requests are no longer counted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CollectInputSystems copied each config field's input systems as they were, so a null tag reached ClassifyByText, where counting its text called Dictionary.TryGetValue(null) and threw. The exception escaped LfWsToLcmWs and ended the whole transfer. An empty tag did no harm but named no writing system either. Both are now left out when the config is read, which ConfiguredTags also goes through. The 2026-10-01 corpus has no such tag in any project's config, so this is only a guard. The new test fails without this, with an ArgumentNullException. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BestStringAndWsId, which the single-string lexicon fields, picture captions and the String custom field go through, took the first non-empty value in document order when two keys named one writing system. WriteToLcm, which writes the multitext fields, takes the key spelled as the project's config spells the writing system, since that is the one LF's editor shows. The two could pick different spellings' text for the same keys, and the single-string fields could pick the export's hidden, stale value over an LF user's edit. BestStringAndWsId now chooses each writing system's key with the same rule, from the same configured spellings, before it leaves out empty values, so a value the user has cleared under the config's spelling is not replaced by the hidden one either. The 2026-10-01 corpus holds no two keys naming one writing system in any single-string field, so this changes nothing there today. The two new tests fail with the old choice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SpanStrToTsString gave a run whose span lang resolves to no LCM writing system
writing system 0, and building the string then threw ArgumentOutOfRangeException
("The specified writing system code is invalid"). Inside WriteToLcm that came
after earlier alternatives had been written and before the clearing pass, so
the entry's per-entry catch left it half-written -- the failure c66141f fixed
for single-string fields whose text could not be placed. 8900058 resolved span
langs through WsIdFromLfTag, which fixed non-canonical spellings of writing
systems LCM has, but not this.
Such a run's text is now kept in the string's own writing system, losing only
its language mark, and a warning names the lang and the text.
In the 2026-10-01 corpus 29 of the 83,743 span langs in Send/Receive projects
name no writing system those projects have, in five projects, and none is in an
entry edited since its project last synced.
The new test fails without this, with the ArgumentOutOfRangeException.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LcmWsToLfWs set each input system's VernacularWS and AnalysisWS flags with the current lists' own Contains, which compares Ids exactly. LCM resolves the lists' Ids ignoring case, and LfTagsOf reads them that way too, so a list spelling an Id in another case put the writing system in the list LF is given but flagged its input system as not in it. testlangproj's vernacular list names "qaa-Zxxx-x-kal-AUDIO" for "qaa-Zxxx-x-kal-audio"; a TODO in TransferLcmToMongoActionTests asked why that one was not flagged, and worked around it. The flags now come from the handles of the writing systems the lists yield, read the same way as LfTagsOf reads them. The TODO and its workaround are gone, and that test now checks every vernacular writing system is flagged. Language Forge does not read the flags, so nothing it shows changes. The new test fails without this: the flag is false. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ResolveByCompanions' summary gave as its example a writing system the field loop has already placed: one sharing a custom field with the vernacular alone. What the pass settles is a writing system offered only in fields that said nothing of their role, whose company there another field has since placed. It also makes one pass, so a writing system whose only company is another one in doubt keeps both roles; the summary now says so. The header of ConvertMongoToLcmWritingSystemsTests said LCM's lookups match an Id literally and that the tests add nothing to the fixture. The lookups ignore case, as the file itself says further down, and many tests now add writing systems of their own, each under a tag no other test uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A single-string field keeps its value when none of LF's text can be placed, and logged that the text "has nowhere to go, since LCM has none" of its writing systems. That is one reason. The other is that the text is all under keys passed over for an empty key naming the same writing system: the config's spelling, cleared by an LF user or filled in empty by LF's editor beside the export's value under the Id. LCM has that writing system, so the log named the wrong cause. Each key with text left out is now reported with its own reason, and the field still keeps its value, since nothing here tells a clear from the editor's empty value. The String custom field, which said nothing at all, reports the same way. The choice of one key per writing system, which WriteToLcm and BestStringAndWsId each made, is now made in one place, ChooseKeys, and KeysPassedOver gives the keys it leaves out. The Mongo test double kept its entries by round-tripping them through LfMerge's serializer, which drops a multitext holding an empty value, so no test could hand the converter LF's empty values. It now keeps entries as documents, as Mongo does; LfMerge's own writes still go through its serializer, as they do against Mongo. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every test of choosing between two keys for one writing system handed LfMultiText a set of configured spellings built by hand. Had ConfiguredTags returned nothing, or a writer stopped being given the set, those tests would still have passed while a sync quietly went back to writing the later key. This test runs a whole sync with the config's spelling added to the field's input systems and put first in the entry, so that going on key order alone writes the export's value instead. It covers the built-in multitext fields (picture captions share their writer), the single-string fields, and the multitext and string custom fields. With ConfiguredTags made to return an empty set, all four cases fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test's writing system, "kal-fonipa", is a phonetic spelling of the vernacular, so language affinity made it vernacular only whether or not it had pronunciation text; with the pronunciation rule removed from ClassifyByText, the test still passed. It now uses "xyz-fonipa", a language the project has nowhere else, and runs both with and without the pronunciation text: vernacular only with it, both and unresolved without. With the rule removed, the case with pronunciation text fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e new MongoToLcm_InputSystemNewToLcm_CountsTheLexiconToo added a single new input system, so it expected two reads of the lexicon whether or not the classification was kept after the first new writing system. Dropping that, and counting the whole lexicon again for every new writing system, would have gone unnoticed. The test now adds two and still expects two reads; without the cache it sees three. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The field loop's comment, and the docstring of the test covering it, said "en" is not made doubtful by turning up in "a note". The entry's note is an analysis anchor, which the loop skips; the field meant, and the one the test uses, is a sense's general note. LogTextNotPlaced's messages are right only when nothing was written, which both its callers ensure but its doc did not say. SetLastSyncedDate's comment still compared it with DeepCopy, which the double no longer uses for entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing defines FW8_COMPAT, and LfMerge has not built against FieldWorks 8 for years, so the #if branches naming IWritingSystemManager and IWritingSystem were never compiled. Only the FieldWorks 9 types are left, and with them goes the comment explaining why the #if blocks were there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LfMerge now exports every writing system by its LCM Id. It rewrites the inputSystems record and custom-field config on every sync, but the built-in fields' config only at initial clone, so projects cloned while LfMerge spelled input systems by LanguageTag keep a config LF's editor cannot match against the lexicon's Id-spelled keys. migrate-input-systems-to-lcm-ids.js re-spells, for the fifteen projects the 2026-07-06 and 2026-10-01 mongodumps show are affected, every config inputSystems array, the inputSystems record and languageCode, each mapping checked against the project's FieldWorks repository. In che-flex, whose lexicon is still keyed by the Id its Chechen writing system had before FieldWorks moved the text to "ce-Latn", it also renames the lexicon's keys in place, leaving entries' dates alone. It is a dry run unless DRY_RUN=false, leaves an inputSystems entry alone where the new spelling is already there, and changes nothing when run a second time. Run it before the new LfMerge build first syncs these projects, and keep them on HOLD from before it runs until that build is deployed: the old build's Replace call would evict or re-register writing systems keyed by Id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmunn
force-pushed
the
bugfix/canonical-writing-system-tags
branch
from
October 6, 2026 02:46
fffeef6 to
499b2ea
Compare
Collaborator
Author
|
I don't think we have CodeRabbit configured for this repo, but in case that's changed: @CodeRabbit fullreview |
rmunn
changed the title
migrate-input-systems-to-lcm-ids.js is run once against Language Forge's Mongo before this build first syncs the projects it lists, and is then of no further use, so it does not belong in LfMerge's long-lived code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Devin raised a few points, all of which were invalid (Claude could see actual data while Devin was just theorizing about it). Both Claude and Devin are now in agreement in finding no more major issues. I will now test the migration script against a dev-machine copy of the repos that need it, and report my results tomorrow. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When LfMerge started supporting FieldWorks 9.x, it started using the
.LanguageTagproperty of writing systems instead of.Idas it had done in FW 8. That proved to be a mistake, which I only caught once I started testing the migration path of non-SR writing systems. Because the.Idproperty is whatever the user entered, whereas the.LanguageTagproperty is a canonicalized form of the tag. And while it seemed like the canonicalized form is what should be used, Language Forge stores writing systems using the ID the user entered. So if the user entered something likeqaa-x-qaa-v, the.LanguageTagwasqaa-x-vinstead, resulting in LfMerge thinking that the ws didn't exist and trying to create it. This then produced errors like "Unable to set writing system 'qaa-x-v' because this id already exists."At first I thought the solution was to canonicalize writing systems before storing them, so that
qaa-x-qaa-vwould disappear. But the LF data still would have hadqaa-x-qaa-vall over the place, and that would have resulted in a massive data migration. So I chose the other option, of having LfMerge stop using.LanguageTagand start using.Idwhen there was a writing system not found in FLEx (which only happens with new S/R projects, since all existing S/R projects had already had writing systems synced up).This does result in about 16 existing S/R projects needing a migration script to be run, but that migration script is a lot smaller than what it would have been if the "canonicalize writing systems before storing them" option had been picked.
LfMerge now has a single helper function,
WsIdFromLfTag, which all uses of writing systems go through. It first looks up the.Idvalue that LF has; if that's not found, then it tries the canonical form. If a project has data in both forms (e.g. if it haden-USanden-Latn-USwriting systems, and both had data) then a warning is logged.Other fixes
A few other bugfixes got folded into this PR as well:
dirtySRfield was null or missing in many non-SR projects, which LfMerge was not designed to handle. LfMerge's Mongo deserializer now treats a null/missingdirtySRfield as 0.#if FW8_COMPATblocks have been removed, since LfMerge no longer supports FW 8 (and in many cases they had become identical to the FW 9 handling anyway, so it would have been#if FW8 doFoo() #else doFoo() #endif, which is just unnecessary.