From daa2b82ee4e86ee8dc63083f0e9d433311aadbba Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Thu, 3 Sep 2026 08:43:39 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20TODO.substantiate=20register=20?= =?UTF-8?q?=E2=80=94=20the=20substantiation=20close-out?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Index + five items. Post-execution state: 01 papers COMPLETE (#80), 02 CE-curves COMPLETE (ml #146), 03 GKD ARMED (ml #147, launch = owner), 04 G2b WATCHING (rung census recorded), 05 closeouts COMPLETE (#38/#41 closed; all 56 Dependabot alerts auto-fixed by #48+#78). --- TODO.substantiate/01-paper-currency.md | 63 +++++++++++++++++++++ TODO.substantiate/02-ce-curve-comparison.md | 25 ++++++++ TODO.substantiate/03-gkd-arming.md | 28 +++++++++ TODO.substantiate/04-g2b-watch.md | 18 ++++++ TODO.substantiate/05-project-closeouts.md | 31 ++++++++++ TODO.substantiate/README.md | 25 ++++++++ 6 files changed, 190 insertions(+) create mode 100644 TODO.substantiate/01-paper-currency.md create mode 100644 TODO.substantiate/02-ce-curve-comparison.md create mode 100644 TODO.substantiate/03-gkd-arming.md create mode 100644 TODO.substantiate/04-g2b-watch.md create mode 100644 TODO.substantiate/05-project-closeouts.md create mode 100644 TODO.substantiate/README.md diff --git a/TODO.substantiate/01-paper-currency.md b/TODO.substantiate/01-paper-currency.md new file mode 100644 index 0000000..74c6598 --- /dev/null +++ b/TODO.substantiate/01-paper-currency.md @@ -0,0 +1,63 @@ +# 01 — Paper currency: the three latex papers vs the Sept sprint + +Priority: P1. Status: OPEN. + +The papers (docs/paper-{arabic,hebrew,umbrella}/main.tex on rababa +main) were last content-audited Aug 19 (commit 7771e8e, "r5 = +2.6775/1.5965 canonical"). The September sprint invalidated large +parts of their tables and framing. This item brings all three +current, number-by-number against docs/RESULTS.md, and recompiles. + +## What changed since Aug 19 (must land in the papers) + +1. **Teachers: r5 -> r6 -> r7.** Canonical Arabic teacher is now r7 + at **2.2864/1.3343** (r6 2.5793/1.5317 was the morph-aux win; r7 + adds the news-domain mix). The umbrella headline still says + 2.68/1.60 (r5). +2. **Client tier: 8.259 -> 4.822 -> 4.5701.** The 2.0 rung (r7 labels + + Muon, E3/E4) and G2a (6 epochs) — 42%+ error reduction at + identical architecture. Paper-B lever table in PUBLICATION-NOTES + is the source. +3. **The GLM family regression axis (complete 2026-09-02):** + GLM-5.2 2.5060 raw / 2.6911 zero-skip -> GLM-5.3-Flash 8.5721 / + 8.7978 (reasoning_effort=low; thinking UNDIS ABLE — 400/1210) -> + GLM-5.3 9.9760 / 9.8971 -> glm-4.7-flash 13.0035 / 13.2256 + (thinking-disabled, 12 passes past 429s). Attribution (rates over + GT-marked positions): wrong-haraqat axis 5.2 2.64% (matches r7's + 2.62%) vs Flash 10.05% / 5.3 8.59% / 4.7 9.01%; missing axis 4.7 + 6.67% worst. Bootstrap CIs in RESULTS.md. Claude-3.7-Sonnet's + 1.3941 (vendor-published) remains the one row ahead of r7. +4. **E5/E6 negatives** (PUBLICATION-NOTES section 8): MTP-aux 5.0853 + (+0.26pp vs 4.8218 control, preemption confound disclosed) and + constant-budget register swap 5.8057 — the data-vs-architecture + pair; levers that moved: optimizer (E3 −2.96pp) >> fresher + teacher labels (−0.47pp) > epochs (G2a −0.25pp... verify from + lever table). +5. **E1 head-fp32 quantization fix** (PUBLICATION-NOTES section 5): + quantized head was the fragility; heb int8 9.34% -> 0.26% flips; + five-of-five rebuilt with flip CIs. +6. **Hebrew: s46 Dicta surfaces 0.5209/0.2377/0.4523** + D-Nikud + publishes NO numbers on those corpora (their Nakdimon-test table + is the vendor row); angle-bracket matres protocol note. Hebrew + paper's modern-text gap framing changes. +7. **Measurement discipline**: bootstrap-CI policy; empty-sentinel + resume guard (15.96 -> 8.57 story); protocol ledger existence. + +## Steps + +- [ ] Read all three main.tex fully; list every number and claim +- [ ] Diff each against docs/RESULTS.md (the ledger) + EXPERIMENTS + + PUBLICATION-NOTES; replace superseded values +- [ ] Weave the new findings into framing (esp. umbrella: the + frontier-regression result is now a headline contribution; + arabic: the lever ladder + negatives; hebrew: Dicta surfaces) +- [ ] Recompile all three PDFs; visual sanity check +- [ ] Commit via PR (docs); the tex trees are tracked on main + +## Sources to read + +- docs/RESULTS.md (ledger + GLM sections + Hebrew Dicta section) +- ml-models docs/PUBLICATION-NOTES.md (sections 5, 8, Paper-B table) +- ml-models docs/paper.adoc (sections already current — 431 lines; + the adoc is ahead of the latex; port from it) +- ml-models docs/EXPERIMENTS.md (E1-E6 gates + verdicts) diff --git a/TODO.substantiate/02-ce-curve-comparison.md b/TODO.substantiate/02-ce-curve-comparison.md new file mode 100644 index 0000000..7ad0220 --- /dev/null +++ b/TODO.substantiate/02-ce-curve-comparison.md @@ -0,0 +1,25 @@ +# 02 — E5/E6 CE-curve comparison vs run-006 at matched steps + +Priority: P1 (quick). Status: OPEN. + +Registered checkbox from TODO.training-work/03+04, never done: the +aux/data levers' effect on convergence speed is itself a finding. + +## Steps + +- [ ] Extract per-step training CE for run-006 (control), E5 + (run-007-r7-muon-mtp), E6 (run-008-tashkeela-mix) at matched + steps — from local run logs (/tmp/e5*.log, /tmp/e6_distill*.log) + and volume step dirs if logs are gone +- [ ] Record a compact matched-steps table in EXPERIMENTS.md E5/E6 + entries (no new plots unless trivial; the table is the record) +- [ ] One-sentence finding in PUBLICATION-NOTES: did either lever + change convergence speed? (E6's register swap at constant + budget: expect similar CE; E5 with the fresh-head confound for + the final 23%: CE jump visible at step ~8,650 — disclose) + +## Caveat + +E5's curve carries the disclosed preemption confound (fresh aux head +from step 8,500; resume loss 0.73 = 0.15*ln(384)+CE) — any CE jump +at that step is the confound, not the lever. diff --git a/TODO.substantiate/03-gkd-arming.md b/TODO.substantiate/03-gkd-arming.md new file mode 100644 index 0000000..c6471a5 --- /dev/null +++ b/TODO.substantiate/03-gkd-arming.md @@ -0,0 +1,28 @@ +# 03 — GKD rung: armed and owner-gated + +Priority: P2. Status: OPEN (arming is mine; LAUNCH IS THE OWNER'S). + +The last registered training lever (SOTA strategy 2025: "GKD is next +lever"). On-policy distillation (student-sampled decoding vs teacher +forcing) targeted at the client tier's domain gap. + +## Arming steps (mine, no launch) + +- [ ] Add `ara-diac-small-2-gkd` spec to distill_specs.yaml: same + control as run-006 (30k units, identical steps/schedule), delta + = GKD loss mixing (on-policy sequences sampled from the student + during training, scored by the frozen r7 teacher) — follow the + implementation notes in PUBLICATION-NOTES section on closing + the domain gap ("on-policy distillation; section-discussion") +- [ ] Pre-register in EXPERIMENTS.md: gate adopt at <= 4.5218 (the + standing E-bar), honest-report band to 4.8218, prediction + 4.3-4.7 (on-policy should attack the domain residual that E6's + swap did not) +- [ ] Unit-level wiring check only (no GPU run): the sampler must use + the STUDENT's current weights, temperature-matched to greedy + inference + +## Launch + +Only on owner instruction (register 05 in TODO.publish). GPU budget: +one A10G-class slot for ~11k steps. diff --git a/TODO.substantiate/04-g2b-watch.md b/TODO.substantiate/04-g2b-watch.md new file mode 100644 index 0000000..01566fc --- /dev/null +++ b/TODO.substantiate/04-g2b-watch.md @@ -0,0 +1,18 @@ +# 04 — G2b verdict watch + cross-recording + +Priority: P2. Status: OPEN (dependent on other agents' run). + +G2b (48k units from full Tashkeela — the ADD direction) is in flight +on the auto-chain. E6's swap-negative makes G2b the decisive test of +the domain-coverage hypothesis: if ADD helps where SWAP hurt, the +residual was budget-limited domain coverage; if ADD also fails, the +residual is not domain-shaped at all. + +## Steps + +- [ ] Watch for final_eval.json under the G2b run dir (checkpoints + volume, spec per ml EXPERIMENTS.md G2b registration) +- [ ] When it lands: verify against its registered gate; record the + E6-swap vs G2b-add pair in PUBLICATION-NOTES Paper-B framing +- [ ] Do NOT duplicate-launch anything (other agents own the run; + modal app list showed their ephemeral apps) diff --git a/TODO.substantiate/05-project-closeouts.md b/TODO.substantiate/05-project-closeouts.md new file mode 100644 index 0000000..9f146c8 --- /dev/null +++ b/TODO.substantiate/05-project-closeouts.md @@ -0,0 +1,31 @@ +# 05 — Project close-outs + +Priority: P3. Status: OPEN. + +## Issue closures (reversible, sanctioned "close superseded") + +- [ ] rababa #38 "Investigate case ending training" — superseded: the + r6 morphological-aux teacher directly targets iʿrāb (case + endings), verified 2.5793 -> r7 2.2864; the finding is recorded + in ml EXPERIMENTS + PUBLICATION-NOTES section on the aux-task + win. Close with that cross-reference. +- [ ] rababa #41 "Farsi" — superseded: Persian models shipped in + interscript-ml (paper.adoc's SentenceBench homograph row); + point to the paper + models.yaml entries. Close. +- [ ] #45 (repo split), #36/#37 (dataset/Farasa comparisons): leave + open — genuine roadmap items, not clearly superseded. + +## Legacy Dependabot alert dismissals (option A) + +- [ ] Dismiss the alerts targeting python/{arabic,hebrew}/ + requirements.txt (reason: vulnerable_code_not_in_use — eager CI + is the live validator; the pins are provenance; TODO.publish/06 + decision record). Keep root-floor alerts open until Dependabot + rescans post-#78. +- [ ] If the API denies permission (needs security-events write), + report — dismissal from the security tab is then the owner's. + +## Register hygiene + +- [ ] After all items in this folder complete, update README.md index + statuses and mark the campaign registers closed in memory. diff --git a/TODO.substantiate/README.md b/TODO.substantiate/README.md new file mode 100644 index 0000000..c889207 --- /dev/null +++ b/TODO.substantiate/README.md @@ -0,0 +1,25 @@ +# TODO.substantiate — the substantiation close-out + +Created 2026-09-03, from the post-campaign inventory. Everything the +work needs to be fully substantiated: papers current, the registered +loose ends closed, owner acts prepared. Campaign registers that are +COMPLETE and closed: TODO.training-work, TODO.publish. + +## Index (priority order) + +| # | Item | Priority | Status | Blocked by | +|---|---|---|---|---| +| [01](01-paper-currency.md) | Three latex papers current with the Sept results | P1 — highest value | COMPLETE (rababa #80) — venue formatting/submission is the owner's | nothing | +| [02](02-ce-curve-comparison.md) | E5/E6 CE-curve vs run-006 at matched steps | P1 — quick, mechanical | COMPLETE (ml #146) | nothing | +| [03](03-gkd-arming.md) | GKD rung: arm spec + registration (launch = owner) | P2 | ARMED (ml #147) — launch on owner word | owner ordering | +| [04](04-g2b-watch.md) | G2b verdict watch + cross-recording | P2 | WATCHING — no final_eval yet as of 2026-09-03; rung census: 8.26/7.56/4.83/5.29/5.08/4.57/5.09/5.81/5.78/73.95 | other agents' run | +| [05](05-project-closeouts.md) | Issue closures #38/#41 + legacy alert dismissals | P3 | COMPLETE — #38/#41 closed with cross-refs; all 56 Dependabot alerts auto-FIXED by #48+#78 on rescan (no dismissals needed) | nothing | + +## Working rules (unchanged) + +- Protocol-matched numbers only; every paper number must trace to + rababa/docs/RESULTS.md (the ledger is ground truth). +- Never fabricate; if a draft cites a superseded number, replace it + with the ledger value — never smooth over. +- Owner-only acts (GKD launch, releases, alert dismissal if API + denies) are prepared, not executed.