[johnml1135]

Stacked on #538 (F4: literal-rewrite unapplication); merge #538 first.

Retargeted to master. The conformance suite (#480) is deferred until after this release, so this PR no longer stacks on it. The conformance figures below were measured while it was stacked and are kept as the record.

Quick summary

HermitCrab now drops reduplication analyses whose copies provably disagree, since synthesis would reject them anyway.
Copies are lined up across unapplied deletions; anything it cannot judge, such as a part the rule modifies, is kept.
Synthesis is untouched; the new Morpher.PruneDisagreeingCopies option defaults to on, covers affix-process,
realizational and compounding rules, and also applies while tracing.

Where to look

• AnalysisMorphologicalTransform.HasDisagreeingCopies / CanAlign -- the invariant: an undecidable copy is always kept,
  and an optional segment may stand for any number of segments it unifies with.
• AnalysisMorphologicalTransformRuleSpec.CopiesMayAgree -- installed as the spec's Pattern.Acceptable, so the rules
  use MultiplePatternRule unchanged.
• PruneDisagreeingCopiesTests -- split selection, on/off pruning, on-by-default, all three rule paths,
  pairwise comparison of three copies, optional segments that do and do not line up, parallelism/memo settings,
  partial morphemes with final templates, tracing, and cap-limited-deletion and counterbleeding-opacity cases where forward generation defines the
  expected parse.
• AffixProcessRuleTests.ReduplicationRules / ModifyFromInputRules -- run with pruning both off and on.

Deliberately not included

• Reruns of the Sena and Amharic parity sweeps (no full-copy rules, so they never enter the new code path).
  The earlier figures are kept below the rule, labelled as prior measurements.

Validation

• dotnet build -c Release -- Build succeeded, 0 Warning(s), 0 Error(s).

• dotnet test -- all four test projects pass; HermitCrab Passed: 131, Failed: 0, Skipped: 0.

• csharpier check . (1.2.6) -- Checked 721 files, clean.

• pwsh -NoProfile -File scripts/comment-hygiene.ps1 -- clean.

• Mutants: accepting every match fails the six pruning tests across affix, realizational, compounding and
  tracing; aligning everything fails OptionalSegmentThatCannotLineUpIsPruned; letting an optional segment
  stand for at most one segment fails CapLimitedDeletionInOneCopyParsesTheSameWithAndWithoutPruning.

• Corpus parity against the previous rule (copies with optional nodes always kept), analyses compared by
  morpheme sequence, 20 s cap per word:

  Grammar	Words compared	Differences	Time
  Indonesian	70	0	4.2 s -> 2.9 s
  Aweti	174	0	281 s -> 25 s
  Aweti words over 20 s before (90 s cap)	14 of 20	0	744 s -> 6.6 s; all 20 now finish in <= 3.3 s
  Mbugwe	1,466 of 1,645	0	1,880 s -> 1,863 s (few copies ruled out)

---

Reading this a year from now

A full-copy reduplication rule (CopyFromInput of the same part twice) is unapplied by matching each
copy as an independent capture; only the first capture rebuilds the base. So every split where the
inserted material lines up becomes its own analysis, and synthesis throws the disagreeing ones away much
later. This change moves that rejection earlier, into analysis, for matches that can be judged.

Synthesis writes every copy of a part from the same input, and anything that later changes one copy is
unapplied before this rule runs, so a match whose copies are proven to disagree segment by segment can
never survive synthesis. An unapplied deletion leaves an optional segment, and with DeletionReapplications
at its default of 0 one optional segment can stand for several deleted ones, so copies are lined up with each
optional segment matching any number of segments it unifies with, including none. A part the rule modifies,
or an optional node that is not a segment, cannot be judged and is never reported as disagreeing -- it is
kept, at the cost of the candidates the prune would otherwise have removed.

Decisions, and why

• The check is the rule spec's Pattern.Acceptable rather than a MultiplePatternRule subclass or a change
  to MultiplePatternRule itself, which is generic SIL.Machine.Rules code. With AllSubmatches = true the
  matcher drops exactly the matches a post-filter would. Affix-process specs install it only when a part is
  copied, since the matcher builds a Match for every candidate it tests.
• Pruning also applies while tracing: a pruned match is a pattern that did not match, which a trace already
  leaves out.
• The classifier was originally a three-state Consistent / Inconsistent / Undecidable enum. No
  caller or test distinguished Consistent from Undecidable -- the pattern rule only ever
  checked for Inconsistent, and both other states took the same keep path -- so it was collapsed to a
  single HasDisagreeingCopies bool, removing the enum and the now-unreachable
  branching that tracked the discarded state.
• MorphologicalOutputAction.GetSkippedOptionalNodes became protected internal static so the copy
  check reuses the same optional-node walk; existing derived callers compile unchanged.
• Copies are compared pairwise, not each against the first, because unifiability is not transitive.
• Morpher.PruneDisagreeingCopies's summary no longer promises a specific memory/timing outcome; the
  contract is what the option controls (skip provably-disagreeing copies) and its default, not a
  particular grammar's measured speedup.
• AnalysisAffixProcessAllomorphRuleSpec and AnalysisCompoundingSubruleRuleSpec take the Morpher as a new
  optional constructor parameter (null means no pruning). Existing source compiles unchanged; callers compiled
  against the old constructors need a rebuild. Nothing in this repository constructs them outside the three
  analysis rules.

Prior measurements (reported on an earlier revision of this branch, not reverified this session)

Check	Result
Aweti index 182, default settings	OOM at 316 s -> 9 s, 2 analyses
Aweti whole list (206)	192 complete (was 172); 4.95x on the 172 both finish; 0 analysis-count differences
Exact analysis-signature parity, off vs on	0 divergences in 1,855 words: Sena 209, Mbugwe 562, Amharic 656, Aweti 179 x 2 exports, Indonesian 70
Real reduplicated analyses	Mbugwe 759/759 accepted copy analyses preserved
Conformance suite (#480, incl. 9 new reduplication x phonology words), prune on	43/44, identical to the branch without this change; the prune fires 612 times
Mutants (prune agreeing copies / undecidable copies / everything)	each fails unit tests and 4-8 conformance words

Grammars without full-copy rules (Sena, Amharic) never enter the new code path. The unpruned parse of
Aweti index 182 never finishes, so its 2 analyses rest on the argument above plus the parity sweep, not a
direct comparison. Mbugwe gains little from this change alone (the prune fires on 161 words but removes
0.2% of candidates).

🤖 Generated with Claude Code

---

This change is 

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.03846% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 74.57%. Comparing base (ebd3a38) to head (c8860f0).

Files with missing lines	Patch %	Lines
...logicalRules/AnalysisCompoundingSubruleRuleSpec.cs	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@                       Coverage Diff                       @@
##           fix/hc-strrep-unapplication     #519      +/-   ##
===============================================================
+ Coverage                        74.46%   74.57%   +0.11%     
===============================================================
  Files                              456      456              
  Lines                            38372    38472     +100     
  Branches                          5271     5297      +26     
===============================================================
+ Hits                             28573    28692     +119     
+ Misses                            8634     8617      -17     
+ Partials                          1165     1163       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[johnml1135]

Pushed 2e1eb06: CopyAgreementPatternRule now computes HasRepeatedParts once in its constructor instead of on every unapplication. Rules that copy nothing no longer rescan their captured parts.

The same change in PanGloss (Rust) was measured against v0.4.0 on words both builds finish, with identical analyses:

Grammar	Fixed / v0.4.0
Aweti	0.62x, 0.64x (two rounds)
Mbugwe	0.97x (two clean rounds)
Amharic	1.01x

All runs used AlwaysEnforceFinalTemplates off. I haven't timed the C# change separately; it removes the same per-call work.

🤖 Generated with Claude Code

[jtmaxwell3]

I don't understand why MultiplePatternRule doesn't already filter this, given that PatternRule does. What is synthesis doing different from analysis? I noticed that AnalysisAffixProcessRule doesn't propagate the SyntacticFeatureStruct through during Apply the way that SynthesisAffixProcessRule does. Is this the issue? Or is the issue that we don't have the stem features during analysis?

If we changed things so that analysis filters the way that synthesis does, that would be a more general solution.

[Update] Synthesize doesn't filter the bad reduplication until the very end, when the output is checked against the original word. So it isn't filtered by PatternRule.

[jtmaxwell3]

I don't know if it is worth it, but I don't think that you need to treat matches with optional nodes as undecidable. Instead, you could non-deterministically match the two lists of nodes. Compare the first nodes. If they match, increment both indices, add 1 to the number of matches, and try again. If either of them is optional, then increment its index and try matching again. When you get to the end of the lists, compare the number of matches against the number of matches allowed by the part (e.g. 2, 0+, 1+, etc.). If at any point the number of matches exceeds the number of matches allowed, then stop.

This is exponential in the number of optional nodes, but you can make it O(n^2 * min(n, max #matches allowed)) if you cache on (index1, index2, #matches).

[jtmaxwell3]

I don't think that it is necessary to turn pruning off when tracing. We don't record rules that don't match when tracing, and pruning reduplication matches is just an extension of this. If Try A Word is too slow, then it isn't useful. I turned MergeEquivalentAnalyses off when tracing because it hides a wide class of failures. I don't think that it is a good analogy for pruning reduplication matches.

[jtmaxwell3]

You no longer need a separate DisagreeingCopiesPruningRule. It appears wherever MultiplePatternRule used to appear. You can move the changes into MultiplePatternRule and delete DisagreeingCopiesPruningRule.

[johnml1135]

I don't understand why MultiplePatternRule doesn't already filter this, given that PatternRule does. [...]
[Update] Synthesize doesn't filter the bad reduplication until the very end, when the output is checked against the original word.

Your update is right: nothing compares the copies until synthesis checks its output against the original word, so PatternRule never sees the disagreement. Checking at match time is the whole of this PR.

[johnml1135]

I don't know if it is worth it, but I don't think that you need to treat matches with optional nodes as undecidable. [...]

It is worth it; 81ff2fe does it. One change from your version: with DeletionReapplications at its default of 0, analysis restores one optional segment where a deletion may have removed several, so an optional segment has to be allowed to stand for any number of segments it unifies with, not at most one. Matching it to at most one pruned the valid parse in CapLimitedDeletionInOneCopyParsesTheSameWithAndWithoutPruning (ggasaasa, where both gs of the second copy were deleted). It's a reachability table over (i, j), so O(n·m) per pair of copies; copies without optional nodes keep the linear comparison, and an optional node that isn't a segment still keeps the match.

Measured against the previous rule (copies with optional nodes always kept), analyses compared by morpheme sequence:

Grammar	Words compared	Differences	Time
Indonesian	70	0	4.2 s → 2.9 s
Aweti	174	0	281 s → 25 s
Aweti words over 20 s before	14 of 20 (the other 6 still exceed 90 s unpruned)	0	744 s → 6.6 s; all 20 now finish in ≤ 3.3 s
Mbugwe	1,466 of 1,645 (the rest exceed 20 s on one or both builds)	0	1,880 s → 1,863 s; few copies are ruled out

[johnml1135]

I don't think that it is necessary to turn pruning off when tracing. [...]

Agreed: a pruned copy match is a pattern that didn't match, and the trace already leaves those out. 425f1f3 removes the tracing exemption; TracingStillPrunesDisagreeingCopies now asserts the match is dropped with tracing on, and the PruneDisagreeingCopies summary no longer mentions tracing.

[johnml1135]

You no longer need a separate DisagreeingCopiesPruningRule. [...]

Done in 425f1f3, through the pattern rather than MultiplePatternRule itself, since that class lives in SIL.Machine.Rules and knows nothing about copies: the check is now the rule spec's Pattern.Acceptable, the hook the compounding spec and the phonological analysis specs already use. All three analysis rules are back to plain MultiplePatternRule, and DisagreeingCopiesPruningRule is deleted. With AllSubmatches = true the matcher drops exactly the matches the old loop dropped.

• Affix-process specs set the predicate only when a part is copied, because the matcher builds a Match for every candidate it tests; other rules pay nothing.
• The compounding spec ANDs it with its existing head-capture check.
• The spec constructors take the Morpher as a new optional parameter (null means no pruning).

Validation: Release build clean; all four test projects pass (HermitCrab 131/131 at 81ff2fe); CSharpier and strict comment hygiene clean. Making the predicate accept every match fails all six pruning tests, which cover the affix, realizational and compounding paths and tracing.

[claude]

1. Verdict: approve. Merge only after Restore analysis of literal rewrites without phonological features #538, which this PR is stacked on.
2. Most important: nothing blocks. F7 at src/SIL.Machine.Morphology.HermitCrab/MorphologicalRules/AnalysisMorphologicalTransform.cs:39 is addressed: the list of parts to check is now built once.
3. Critical 0, Important 0, Low 0 new. F7 is addressed.
4. The compare API reports ahead from 394e730, so I reviewed only git diff 394e7303 c8860f0c, a single-file refactor that keeps behavior the same. The only in-tree reader of the CapturedParts dictionary is AnalysisMorphologicalTransformRuleSpec.cs:46, and it only reads, so the list built in the constructor cannot go stale. Ran dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests -c Release on the checked-out merge: Passed 157, Failed 0. CI Build on ubuntu-22.04 and Build on windows-latest are pending in gh pr checks 519.
5. Not verified: I did not run ./local_check.sh --agent-strict. The Comment hygiene check is green, but it is advisory and does not count as a pass. I did not run the F2 mutants or the corpus timings in the PR body.

Public API: one addition, Morpher.PruneDisagreeingCopies, on by default. Target framework, packages, and parity with sillsdev/machine.py: none verified.

Findings: F1 addressed (e55c37b); F2 addressed (06d2782), mutants unverified; F3 addressed (68700d2); F4 addressed (76cbee7, via #538); F5 addressed (d55b65a); F6 addressed (394e730); F7 addressed (c8860f0).

Reviewed at c8860f0