Decide #236: a MIM: subject is the escaped file stem, and nothing re-derives it

[realmarcin]

Closes #236.

The decision

A MIM: subject is mim_curie_for_stem(<per-record file stem>) — fixed for the life of the record, never re-derived from preferred_term. Recorded in MAPPING_SEMANTICS.md §2 → Subject identifiers.

#236 framed this as a trade-off between stable per-record paths and slug/path agreement. That trade-off only exists if the slug comes from the label. FilenameIndex never renames a file, so a subject derived from the stem gets both.

It was already the rule — it just wasn't written where anyone could find it

claw's publisher	subject_id MIM:<safe_stem> -- stable per-YAML CURIE
CurieNormalizer	known set built from file stems; anything else → UNKNOWN_SUBJECT
MAPPING_SEMANTICS.md	already said "Keep the SSSOM subject_id" — inside a blockquote about spelling corrections

Option 2 was wrong on its own terms. A subject naming no file is not harmless: the resolver returns UNKNOWN_SUBJECT and equivalent_term refuses to cite the mapping.

The 18 "unresolved subjects" were a checker bug

check_sssom_subject_files.py compared subjects against unescaped stems. All 18 were subjects whose stem holds (, ) or α:

file '(R)-lactate.yaml'  ->  MIM:~28R~29-lactate

Measured with the escaping the resolver itself uses: zero genuine mismatches. The tool declaring the question undecided was producing the evidence that it was. It now escapes, and gates — in just qc-sssom and in the qc-sssom workflow.

What still disagreed with the rule

The mechanism #236 was filed about was still live. relabel_mapped_record rewrote cols[0] = f"MIM:{sanitize_filename(new_term)}" on every rename while leaving the file where it was. Its own docstring names the exact record #236 cites. A relabel now changes subject_label only.

merge_salt_label_duplicates had a trap. It dropped f"MIM:{lose_label.replace(' ', '_')}", which only matched because its three labels equal their stems. Switching to the stem needed care: the loser is rewritten to carry the winner's identifier before the drop, and FilenameIndex matches identifier first — so a lookup at that point returns the winner's stem and would drop the winner's rows. The stem is now captured before the rewrite, pinned by a test that demonstrates the trap directly.

kind	sites
label-derived subject	relabel_mapped_record, merge_salt_label_duplicates, reground_aromatic_hydrocarbon (now refuses to guess)
right stem, no escape	create_records_from_groundings*, add_culturemech_gap_labels, move_mapped_out_of_unmapped_collection, promote_resolved_unmapped, promote_microbedecoder_residual, promote_microbedecoder_reviewed
hand-written copy of the escape rule	validate_sssom_invariants, backfill_sssom_surface_forms, build_curie_alias_map, check_sssom_subject_files
dead code rewriting the published SSSOM	decompose_py_media_and_ground_categories (removed)

* compared the unescaped subject against published, so an escape-needing stem would miss the dedupe and emit a duplicate row.

Five copies of the escape rule, and three had independently rediscovered the same bug — the validator's and backfill's docstrings both record it. The fourth copy still had it. All now call mim_curie_for_stem. The validator's copy said it was inlined "to keep the validator stdlib-only"; that stopped being true when Rule J added a package import, and curie.py is stdlib-only anyway.

Rule B1 is unaffected. 23 record stems need escaping; none publishes a narrowMatch, and the registry slug drops those characters rather than escaping them (MIM:Synthetic_Sea_Salts_~28sss~29 ↔ kgmicrobe.compound:synthetic_sea_salts_sss). §2's claim that the two slugs are "always the same" is corrected to say so.

Verification

Canaried in CI's exact environment — Python 3.13 with only pyyaml, package confirmed not importable — with each step's own exit code:

validate_published_curies.py     exit=0
validate_sssom_invariants.py     exit=0
reconcile_sssom.py               exit=0
check_sssom_subject_files.py     exit=0

Every guard mutation-checked by reintroducing its defect:

A relabel rewrites subject again       1 failed
B writer spells subject by hand        1 failed
C second copy of the escape idiom      1 failed
D merge drops a label-derived subject  1 failed
E checker back to raw stems            1 failed
restored                               13 passed

gate	result
qc-sssom (now incl. the subject gate)	OK
qc-flat-coverage, check-unified-freshness	OK
lint / format-check / typecheck / check-instruction-refs	clean
all 15 touched scripts import	OK
unused-name findings in touched scripts	11 on main, 11 here — none introduced

A correction to my own recommendation

The inventory I gave before starting was cut short by a head -10 and missed relabel_mapped_record — the one that matters most. The regression guard scans the whole tree rather than trusting a list.

Also worth knowing: just lint only checks src/, so unused names in scripts/ accumulate unseen — there are 11 pre-existing in these files alone. That's #534's territory.

Full suite is running; I will confirm before asking for the merge.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AcKSD9k5BBntNUFyG3E9H9

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[realmarcin]

Adversarial review of this branch, read-only. Three findings, filed and fixed here.

#689 — I put a stale, mischaracterised number into the spec

I copied "205 renames to date" from curie.py's docstring into MAPPING_SEMANTICS.md as evidence that deliberate renames still happen. mappings/mim_curie_aliases.tsv holds 196 rows, and quoting a mutable count is exactly what CLAUDE.md rules out.

The characterisation was the bigger problem. Classifying all 196:

what changed	rows
escaping only — same stem, now ~HEX	11
case only	68
a different stem	117

And the first "different stem" entry is MIM:2-phenylethylamine → MIM:Phenethylamine_Hydrochloride — the #236 case itself. The alias map hasn't been recording file moves; it has been quietly absorbing the subject-derivation drift this PR stops. Both texts now say what the map is for, with no number.

#690 — "nothing to drop" was asserted, not checked

When FilenameIndex found no stem, the merge script printed that the loser "published no SSSOM row to drop". The index misses a record whose collection copy has drifted from its file on both keys; in that case a real row survives as an ORPHAN while the only message that could have flagged it certifies it instead. That is the #306/#687 shape again.

It now looks for published rows carrying the loser's label and refuses, naming them, if any exist. Mutation-checked: restoring the old message fails both tests.

#691 — the guard exempted whole files

Each of the four exempt files holds exactly one legitimate f"MIM:{...}", but a second, wrong one added later would have sailed through. Exemptions are now line-level by content, and a test fails if an exemption stops matching anything. Mutation-checked: a second bad subject in a formerly exempt file is caught.

Also worth recording

I had two full-suite runs going in the same worktree, and changed files under the older one mid-run — so its result would have described neither version. I stopped it; the one running now is on the final tree.