[johnml1135]

Summary

This adds a conformance suite of synthetic HermitCrab grammars, with CI gates for parse results, coverage claims, and generated ledgers. This revision adds named parser-settings variants. The runner creates a fresh morpher for each run, applies its settings from constructor defaults, and compares complete parse-identity multisets and statuses against that run's expectations. A variant supplies the same ordered words as the default, with complete expectations of its own. Diagnostics name the variant; variants add no fixture or semantic-coverage credit.

Supported settings are DelReapps, MaxRoots, MakePartialsFinal, MergeAnalyses, and explicit per-call GuessRoots. The main review risk is accidentally claiming support for a setting an adapter cannot apply, or treating incomplete output as a valid empty result. Unsupported settings, missing rows, and zero-comparison variants fail; PanGloss also rejects capped or timed-out output.

• PROTOCOL.md, the schema, and WordsYamlLoader: closed settings and strict structure; the loader also checks name uniqueness and ordered inputs.
• WordsYamlMorpherSettingsTests: malformed and mismatched inputs, wrong variant expectations, and variants whose guesses leave no comparable rows.
• Five selected fixtures exercise all supported settings; each engine compares all expected identities and statuses for every run.
• Coverage credit and manifest fixture counts remain independent of variant runs.

No parser engine changed. MaxAlternatives remains unsupported because PanGloss does not implement HC's typed overflow behavior; MergeMSAs remains unsupported because it requires the FieldWorks project loader. MergeAnalyses currently verifies output preservation only; it does not measure merge-fold counts. The new C# and PanGloss contract is documented in Machine's variant protocol.

The PR targets master; review freshly fetched origin/master at 2f70c04698e13f7cf942e4245ae20d77de19ee5a (merge base a2f80e28debaffda8a65b3b11875c80479768bc7). Head: 184bcd46622571a06f1a1fdbdff2fb155714a1c6. The companion PanGloss adapter tested at e2627c9 is on feat/conformance-settings-variants.

Validation at the pinned head: local_check.sh --agent-strict passed; Release had 0 warnings/errors; HermitCrab 634 passed, SIL.Machine 887 passed / 3 skipped, Thot 93 passed, and SentencePiece 4 passed. All 72 C# runs across 62 fixtures passed. The selected five-fixture matrix passed 15/15 runs and 81 word-run rows in both C# HC and PanGloss. PanGloss harness tests: 53 passed; pinned parser tests: 117 passed / 10 marked skipped. The all-scope pin gate passed across 130 fixtures, 1,414 default words, and 10 variants, with three existing pathological/crash exclusions. Broader Rust workspace and Foma suites were not run; production parser engines are unchanged.

Oracle observations and verification detail

C# HC observations were collected on 2026-10-22 with harness draft 1836d23ccb86f1e9da123a43feb661ea76e85d0b, atop the then-current PR #480 head 72dbd4f9bd686d1193d11cf8f908d50ffa878ba1; the HC engine was unchanged. The assembly SHA-256 was 2547068882BEC56873F0678922D11D7F81ED4D68B3B99DCDB71EBF5863714444. The five measured fixture/settings pairs are deletion reapplications (DelReapps), compounding breadth (MaxRoots), final-template partial discriminators (MakePartialsFinal), final-template merge key (MergeAnalyses), and guess-roots settings (GuessRoots). Exact witness outputs and reproduction commands are in the durable observations.

The adapter matrix used the same five fixtures, each with its default plus two variants: 15 complete runs and 81 word-run rows per engine. C# self-check included pathological fixtures. PanGloss validation also included the managed pg-conformance-fixtures and pg-parse checks (both exit 0) and the 130-fixture pin gate. The C# coverage report passed with 31/31 constructs and 15 allowed dead rules. CSharpier checked the 23 changed C# files explicitly. The variant protocol forbids capped or timed-out output from matching an expected empty result; the PanGloss tests include that negative control.

Deferred suite work and prior stack notes

Deferred until after the Hermit Crab Optimizations release. Nothing stacks on this PR any more: #534, #540, #522 and #519 were retargeted to master, and #543 was closed. Their suite-side work is held back and will be added here when this resumes, each once its own PR has merged:

• Trace HermitCrab failure reasons and template slot outcomes #534: FailureRuleAttributor credits through Trace.Allomorph; FieldWorks-producibility rows for the four new failure reasons.
• Fix narrowing-rule unapplication across an interposed optional node #540: the edge-cases/interposed-optional-multi-rhs fixture.
• Remove HermitCrab's analysis-cascade memo #522: removing the harness's --no-memoization flag along with the memo.
• Add conformance fixtures for the #504 and #507 evidence gaps #543: the syntactic-feature-inversion-controls (HermitCrab: track exact analysis feature inversion and chained override parse loss #504) and final-template-gap-coverage (HermitCrab: complete cross-engine correctness controls for final-template pruning #507) fixtures.

The earlier body also recorded these stack descriptions:

• Trace HermitCrab failure reasons and template slot outcomes #534 restores failure-allomorph attribution through its own Trace.Allomorph and classifies its four new FailureReason values in the FieldWorks-producibility ledger.
• Fix narrowing-rule unapplication across an interposed optional node #540 adds interposed-optional-multi-rhs, red on master and green with the fix.
• Remove HermitCrab's analysis-cascade memo #522 removes the analysis memo and the harness's --no-memoization flag.
• Prune reduplication matches whose copies cannot agree #519 was rebased only.
• Add conformance fixtures for the #504 and #507 evidence gaps #543 adds fixtures for the HermitCrab: track exact analysis feature inversion and chained override parse loss #504 and HermitCrab: complete cross-engine correctness controls for final-template pruning #507 evidence gaps.

Suite review map

Where to look

• .github/workflows/conformance.yml -- the gate. Runs on any change under src/SIL.Machine.Morphology.HermitCrab*, src/SIL.Machine, the HermitCrab tests or conformance/: self-check of every fixture, the coverage report, and a freshness check on the generated CSVs.
• conformance/dead-rules-allowed.tsv -- the coverage report now exits 1 on a rule no word exercises unless it is listed here with a reason, and on a listed rule that is exercised again. Today: 15 entries, all deliberate (inactive decoys, refused controls, crash pins).
• conformance/PROTOCOL.md -- the adapter contract a second engine implements.
• src/SIL.Machine.Morphology.HermitCrab.Conformance/Runner.cs -- self-check: fixtures run in-process against HermitCrab as the oracle.
• global.json -- new: pins the SDK to 10.0.100 with latestFeature roll-forward, for the whole repository.

Reading order

About 25k lines are harness code (src/SIL.Machine.Morphology.HermitCrab.Conformance) and 16k are its tests. The rest is data: fixtures (conformance/edge-cases, conformance/languages, about 21k), generated ledgers and evidence cards (about 28k), and docs. Generated files are rewritten by hc-conformance --write-* and pinned by tests, so they need checking for freshness, not reading line by line.

Earlier suite revision and validation decisions

• Rebased onto master (be5407d).
• Dropped Trace.FailureAllomorph: an engine API change does not belong in a test-suite PR, and Trace HermitCrab failure reasons and template slot outcomes #534 adds the same fact as Trace.Allomorph. Generated CSVs regenerate byte-identical without it.
• The coverage report could not fail; it now gates dead rules and uncovered constructs.
• ci.yml is back to master. Its added fixture run duplicated ConformanceFixtureGateTests under dotnet test, and parity-check.py's remaining check is now enforced by the coverage report.
• The counterfactual sweep read mutant output in the console code page, so "katibɯd" came back as "katib?d" and a mutation that changed nothing looked like evidence. It now reads UTF-8 (CounterfactualGate, plus a regression test). Regenerating the ledger demoted two surfaces that had only that false evidence: Gloss to Unobservable and HeadFeatures to RequiredByDtd. The gap ratchet moves from 25 to 26 for Gloss.
• local_check.sh ran --semantic-coverage, which exits 1 by design while the catalog is incomplete, so it could never pass. It now runs the same gates as conformance.yml.
• chained-output-feature-override-loss said it fails on master on purpose. It has passed since Change Add to PriorityUnion #494; checked by running it at 52d069f (red) and 7109531 (green). The header now says so.

Earlier decisions, and why

• PCRE.NET (now on master) ships an IIncrementalGenerator. The compilation-graph loader admits it only by content hash, and a probe runs it and fails closed on any generated output.
• An expect_crash fixture passes only if the same engine completes an ordinary fixture in the same run, so a broken adapter command cannot pass as a crash (conformance/PROTOCOL.md).
• *.fwdata is marked binary in .gitattributes; its pinned hash depended on core.autocrlf.
• Hand-maintained fixture totals were removed from the README and docs; the manifest is the count.
• Four fixture grammars keep trailing whitespace: their hashes are pinned in the manifest.

🤖 Generated with Claude Code

---

This change is

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.93%. Comparing base (2599204) to head (fa8d9b2).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #480      +/-   ##
==========================================
+ Coverage   74.57%   74.93%   +0.36%     
==========================================
  Files         453      556     +103     
  Lines       38314    56481   +18167     
  Branches     5280     7714    +2434     
==========================================
+ Hits        28573    42325   +13752     
- Misses       8576    12291    +3715     
- Partials     1165     1865     +700     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[johnml1135]

Added two independently justified conformance fixtures in a20bce1: final-template-partial-discriminators (10 words; #507) and template-category-sharing (6 words; #505). Fresh Release build: zero warnings/errors. Both fixtures pass with memoization on and off, zero skipped; 22 focused manifest/coverage tests pass. Four grammar counterfactuals produce the independently predicted mismatches, documented under conformance/docs. These are grammar-sensitivity controls, not proof that pruning, cache hits, or same-key feature collisions execute. No parser code changed. Generated inventory also restores previously omitted rows for the already-existing Exact-inverse fixture; its valid zudiua loss remains tracked in #504, not relabeled as correct.

[johnml1135]

Added the shared-template unconstrained-suffix conformance fixture in f150e2a. The grammar has a slot-only feature-unconstrained suffix shared by mandatory TV/IV templates, an ordinary N-to-IV verbalizer, four positive rows, and six rejection controls. The companion rationale explains complete forward derivations and loader reachability (not a FieldWorks UI round trip).

Fresh C# harness: all 10 rows pass in both template orders, memoization on/off (40 comparisons; no skips). Focused template/manifest tests: 22 passed, no skips. Regenerated manifest is current at 40 fixtures; coverage/rule reports updated. No Machine parser code changed.

PanGloss main now contains the narrow gate-only alignment (fc357daf), with this fixture mirrored unchanged until its old Machine submodule pin is upgraded. Direct state assertions provide red/green evidence; full-parse fixture checks pass both before and after the change. Issue #505 remains open for collision/correlation and broader merge-equivalence evidence.

[claude]

FYI: F29. A rule set that was only traced is reported as declared rules. If a fixture declares [crLeftHead] and the engine traces [crWrongHead], the detail includes declared rules [crWrongHead] count 0 != traced count 1, which points the author at the wrong side of words.yaml. Label the key rules [...]: declared N, traced M; ConformanceHarnessTests.cs:97 pins the current wording.

[claude]

FYI: F30. the silent16 is not defined anywhere in the tree, and no assertion checks the 48 positive-parse tags figure, so a reader cannot connect the 1428/799/617 pins to this comment. Name the 16 fixtures or the property they share, or drop the comment.

[claude]

F30 still applies at fa8d9b2: the comment was updated to "132 fixtures" but still ends "across the silent16", which nothing in the tree defines, and nothing checks the 48-tag figure against the new 1430/619 pins.

[claude]

FYI: F31. This comment narrates the corpus delta (574 more than the prior 62-fixture set), which .claude/skills/code-comments/SKILL.md bans as history; it goes stale at the next graduation. The same pattern is at GrammarCoverageLedgerTests.cs:38, DataflowObligationLedgerTests.cs:42 and OrderingGeneratorTests.cs:12. State what the pin measures, or leave the delta to the commit message.

[claude]

F31 still applies at fa8d9b2, and the comment is now wrong: it says 1200 pairs is "574 more than the prior 62-fixture set", but 1191 - 574 = 617 at the last head, so the delta is now 583. That is the staleness this finding predicted. State what the pin measures and drop the delta.

[johnml1135]

Added complete a/ga derivations in a48b899: two a parses and five ga parses, with repeated signatures preserved. The grammar is unchanged. The fixture note records the exhaustive forward chains, full source identities, observed morph order, and provenance.

This revision deliberately exposes #506 and depends on #556. Normal ./local_check.sh --agent-strict ran every stage. It failed only the aggregate fixture self-check and conformance CLI on missing a/ga morph identities from the unfixed engine. Core: 890 passed / 3 existing skips; HC: 695 passed / 1 intended regression; Thot: 93 passed; SentencePiece: 4 passed. Formatting, Release build, coverage and freshness passed. This is not a green merge-ready head.

With the #480 HC/adapter and the exact corrected #556 Machine assembly (d98d12a5), the official full conformance CLI including pathological fixtures passed 142/142 runs across 132 fixtures, zero failures/skips. Complete target identity multisets also matched across ten fresh corrected-engine loads.

The unpatched counterfactual sweep produced false evidence by comparing randomly incomplete morph identities. Corrected-engine reruns over all eleven fixture words (ten fresh processes per mutation) rejected apparent effects from Gloss, empty HeadFeatures, either adjacent rule swap, and lexical/rule POS removal. Only the genuine unordered-to-linear witness was retained. Dependent ledgers and claim-count pins were refreshed; 31 focused ledger/manifest tests and 13 semantic-audit tests passed.

No parser engine code is changed in this fixture commit. #554 and #555 remain separate follow-ups.

Prepared and verified with OpenAI Codex.

[claude]

Major: F32. The conformance gate fails at this head, and the result changes from run to run. The a/ga rows expect parses that only the unmerged #556 produces. CI Conformance suite (job 120631053352): a got [++|a | |a] and ga lost a +++|ga. Locally, EveryConformanceFixturePassesSelfCheck failed with only ga wrong (+|ga). The second Conformance suite run passed by chance. Land #556 first, or keep these rows out until it merges.

[claude]

FYI: F33. This comment pins a review-round commit, "At this #480 head (fa8d9b2...)". That commit is not this head, and it means nothing after merge. code-comments excludes this kind of history. Say "until #506 is fixed" and leave out the SHA.

[claude]

Request changes. Follow-up on fa8d9b2. The branch moved forward by one commit, a48b899, which adds the a/ga truncation rows and regenerates the ledgers, so I reviewed fa8d9b2..a48b899.

1. Verdict: request changes. The conformance gate fails at this head, and the result changes between runs.
2. Most important: F32, conformance/edge-cases/truncate-morphotactic/words.yaml:157. The new rows expect parses that only the unmerged Fix directional annotation tie boundaries #556 produces. Your comment says this is deliberate, but while it stays, EveryConformanceFixturePassesSelfCheck and Conformance suite fail at random on master. Land Fix directional annotation tie boundaries #556 first, or hold these rows until it merges.
3. Counts: 1 Critical new (F32), 1 Low new (F33), 3 Low open (F29, F30, F31).
4. Ran: gh pr checks 480: one Conformance suite run fails (truncate-morphotactic 2/11 mismatched) and the other passes. Both Comment hygiene runs pass. Build on ubuntu-22.04, Build on windows-latest and claude-review are pending. Locally, dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests -c Release --filter ConformanceFixtureGate|ConstructClaimCorroborationTests failed 1 of 11 (ga only, with a different wrong multiset from CI). I read the delta inline and did not run /code-review on it.
5. Could not verify: the regenerated semantic-coverage-counterfactuals.tsv row (truncate-morphotactic as an unordered witness) and the interface-witness counts come from the patched Fix directional annotation tie boundaries #556 engine, by your account. I did not check whether recomputing them on this head reproduces them. I did not run ./local_check.sh --agent-strict.

Public API, target framework, package, or sillsdev/machine.py parity: None verified.

Finding	Severity	Status
F1	Critical	addressed (cc947db)
F2	Important	addressed (bcb3d62)
F3	Important	addressed (bcb3d62)
F4	Important	accepted, unverified
F5	Low	addressed (bcb3d62)
F6	Low	addressed (067de6f)
F7	Low	addressed (0a1212b)
F8	Low	addressed (067de6f)
F9	Low	addressed (067de6f)
F10	Important	addressed (cd0c909)
F11	Low	accepted, unverified
F12	Low	addressed (cd0c909)
F13	Low	addressed (0a1212b)
F14	Low	addressed (0a1212b)
F15	Low	addressed (067de6f)
F16	Low	addressed (067de6f)
F17	Low	addressed (0a1212b)
F18	Low	addressed (0a1212b)
F19	Critical	addressed (e7fe733)
F20	Low	addressed (8b5d74e)
F21	Low	addressed (1c5c3a5)
F22	Low	addressed (a3b4c0e)
F23	Important	addressed (49ba481)
F24	Important	addressed (49ba481)
F25	Low	addressed (9671b28)
F26	Low	addressed (641f613)
F27	Low	addressed (8b5d74e)
F28	Critical	addressed (dd5f00f, 2cd68e8)
F29	Low	open
F30	Low	open
F31	Low	open
F32	Critical	new
F33	Low	new

Reviewed at a48b899