Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions TODO.substantiate/01-paper-currency.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# 01 — Paper currency: the three latex papers vs the Sept sprint

Priority: P1. Status: OPEN.

The papers (docs/paper-{arabic,hebrew,umbrella}/main.tex on rababa
main) were last content-audited Aug 19 (commit 7771e8e, "r5 =
2.6775/1.5965 canonical"). The September sprint invalidated large
parts of their tables and framing. This item brings all three
current, number-by-number against docs/RESULTS.md, and recompiles.

## What changed since Aug 19 (must land in the papers)

1. **Teachers: r5 -> r6 -> r7.** Canonical Arabic teacher is now r7
at **2.2864/1.3343** (r6 2.5793/1.5317 was the morph-aux win; r7
adds the news-domain mix). The umbrella headline still says
2.68/1.60 (r5).
2. **Client tier: 8.259 -> 4.822 -> 4.5701.** The 2.0 rung (r7 labels
+ Muon, E3/E4) and G2a (6 epochs) — 42%+ error reduction at
identical architecture. Paper-B lever table in PUBLICATION-NOTES
is the source.
3. **The GLM family regression axis (complete 2026-09-02):**
GLM-5.2 2.5060 raw / 2.6911 zero-skip -> GLM-5.3-Flash 8.5721 /
8.7978 (reasoning_effort=low; thinking UNDIS ABLE — 400/1210) ->
GLM-5.3 9.9760 / 9.8971 -> glm-4.7-flash 13.0035 / 13.2256
(thinking-disabled, 12 passes past 429s). Attribution (rates over
GT-marked positions): wrong-haraqat axis 5.2 2.64% (matches r7's
2.62%) vs Flash 10.05% / 5.3 8.59% / 4.7 9.01%; missing axis 4.7
6.67% worst. Bootstrap CIs in RESULTS.md. Claude-3.7-Sonnet's
1.3941 (vendor-published) remains the one row ahead of r7.
4. **E5/E6 negatives** (PUBLICATION-NOTES section 8): MTP-aux 5.0853
(+0.26pp vs 4.8218 control, preemption confound disclosed) and
constant-budget register swap 5.8057 — the data-vs-architecture
pair; levers that moved: optimizer (E3 −2.96pp) >> fresher
teacher labels (−0.47pp) > epochs (G2a −0.25pp... verify from
lever table).
5. **E1 head-fp32 quantization fix** (PUBLICATION-NOTES section 5):
quantized head was the fragility; heb int8 9.34% -> 0.26% flips;
five-of-five rebuilt with flip CIs.
6. **Hebrew: s46 Dicta surfaces 0.5209/0.2377/0.4523** + D-Nikud
publishes NO numbers on those corpora (their Nakdimon-test table
is the vendor row); angle-bracket matres protocol note. Hebrew
paper's modern-text gap framing changes.
7. **Measurement discipline**: bootstrap-CI policy; empty-sentinel
resume guard (15.96 -> 8.57 story); protocol ledger existence.

## Steps

- [ ] Read all three main.tex fully; list every number and claim
- [ ] Diff each against docs/RESULTS.md (the ledger) + EXPERIMENTS +
PUBLICATION-NOTES; replace superseded values
- [ ] Weave the new findings into framing (esp. umbrella: the
frontier-regression result is now a headline contribution;
arabic: the lever ladder + negatives; hebrew: Dicta surfaces)
- [ ] Recompile all three PDFs; visual sanity check
- [ ] Commit via PR (docs); the tex trees are tracked on main

## Sources to read

- docs/RESULTS.md (ledger + GLM sections + Hebrew Dicta section)
- ml-models docs/PUBLICATION-NOTES.md (sections 5, 8, Paper-B table)
- ml-models docs/paper.adoc (sections already current — 431 lines;
the adoc is ahead of the latex; port from it)
- ml-models docs/EXPERIMENTS.md (E1-E6 gates + verdicts)
25 changes: 25 additions & 0 deletions TODO.substantiate/02-ce-curve-comparison.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# 02 — E5/E6 CE-curve comparison vs run-006 at matched steps

Priority: P1 (quick). Status: OPEN.

Registered checkbox from TODO.training-work/03+04, never done: the
aux/data levers' effect on convergence speed is itself a finding.

## Steps

- [ ] Extract per-step training CE for run-006 (control), E5
(run-007-r7-muon-mtp), E6 (run-008-tashkeela-mix) at matched
steps — from local run logs (/tmp/e5*.log, /tmp/e6_distill*.log)
and volume step dirs if logs are gone
- [ ] Record a compact matched-steps table in EXPERIMENTS.md E5/E6
entries (no new plots unless trivial; the table is the record)
- [ ] One-sentence finding in PUBLICATION-NOTES: did either lever
change convergence speed? (E6's register swap at constant
budget: expect similar CE; E5 with the fresh-head confound for
the final 23%: CE jump visible at step ~8,650 — disclose)

## Caveat

E5's curve carries the disclosed preemption confound (fresh aux head
from step 8,500; resume loss 0.73 = 0.15*ln(384)+CE) — any CE jump
at that step is the confound, not the lever.
28 changes: 28 additions & 0 deletions TODO.substantiate/03-gkd-arming.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# 03 — GKD rung: armed and owner-gated

Priority: P2. Status: OPEN (arming is mine; LAUNCH IS THE OWNER'S).

The last registered training lever (SOTA strategy 2025: "GKD is next
lever"). On-policy distillation (student-sampled decoding vs teacher
forcing) targeted at the client tier's domain gap.

## Arming steps (mine, no launch)

- [ ] Add `ara-diac-small-2-gkd` spec to distill_specs.yaml: same
control as run-006 (30k units, identical steps/schedule), delta
= GKD loss mixing (on-policy sequences sampled from the student
during training, scored by the frozen r7 teacher) — follow the
implementation notes in PUBLICATION-NOTES section on closing
the domain gap ("on-policy distillation; section-discussion")
- [ ] Pre-register in EXPERIMENTS.md: gate adopt at <= 4.5218 (the
standing E-bar), honest-report band to 4.8218, prediction
4.3-4.7 (on-policy should attack the domain residual that E6's
swap did not)
- [ ] Unit-level wiring check only (no GPU run): the sampler must use
the STUDENT's current weights, temperature-matched to greedy
inference

## Launch

Only on owner instruction (register 05 in TODO.publish). GPU budget:
one A10G-class slot for ~11k steps.
18 changes: 18 additions & 0 deletions TODO.substantiate/04-g2b-watch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# 04 — G2b verdict watch + cross-recording

Priority: P2. Status: OPEN (dependent on other agents' run).

G2b (48k units from full Tashkeela — the ADD direction) is in flight
on the auto-chain. E6's swap-negative makes G2b the decisive test of
the domain-coverage hypothesis: if ADD helps where SWAP hurt, the
residual was budget-limited domain coverage; if ADD also fails, the
residual is not domain-shaped at all.

## Steps

- [ ] Watch for final_eval.json under the G2b run dir (checkpoints
volume, spec per ml EXPERIMENTS.md G2b registration)
- [ ] When it lands: verify against its registered gate; record the
E6-swap vs G2b-add pair in PUBLICATION-NOTES Paper-B framing
- [ ] Do NOT duplicate-launch anything (other agents own the run;
modal app list showed their ephemeral apps)
31 changes: 31 additions & 0 deletions TODO.substantiate/05-project-closeouts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# 05 — Project close-outs

Priority: P3. Status: OPEN.

## Issue closures (reversible, sanctioned "close superseded")

- [ ] rababa #38 "Investigate case ending training" — superseded: the
r6 morphological-aux teacher directly targets iʿrāb (case
endings), verified 2.5793 -> r7 2.2864; the finding is recorded
in ml EXPERIMENTS + PUBLICATION-NOTES section on the aux-task
win. Close with that cross-reference.
- [ ] rababa #41 "Farsi" — superseded: Persian models shipped in
interscript-ml (paper.adoc's SentenceBench homograph row);
point to the paper + models.yaml entries. Close.
- [ ] #45 (repo split), #36/#37 (dataset/Farasa comparisons): leave
open — genuine roadmap items, not clearly superseded.

## Legacy Dependabot alert dismissals (option A)

- [ ] Dismiss the alerts targeting python/{arabic,hebrew}/
requirements.txt (reason: vulnerable_code_not_in_use — eager CI
is the live validator; the pins are provenance; TODO.publish/06
decision record). Keep root-floor alerts open until Dependabot
rescans post-#78.
- [ ] If the API denies permission (needs security-events write),
report — dismissal from the security tab is then the owner's.

## Register hygiene

- [ ] After all items in this folder complete, update README.md index
statuses and mark the campaign registers closed in memory.
25 changes: 25 additions & 0 deletions TODO.substantiate/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# TODO.substantiate — the substantiation close-out

Created 2026-09-03, from the post-campaign inventory. Everything the
work needs to be fully substantiated: papers current, the registered
loose ends closed, owner acts prepared. Campaign registers that are
COMPLETE and closed: TODO.training-work, TODO.publish.

## Index (priority order)

| # | Item | Priority | Status | Blocked by |
|---|---|---|---|---|
| [01](01-paper-currency.md) | Three latex papers current with the Sept results | P1 — highest value | COMPLETE (rababa #80) — venue formatting/submission is the owner's | nothing |
| [02](02-ce-curve-comparison.md) | E5/E6 CE-curve vs run-006 at matched steps | P1 — quick, mechanical | COMPLETE (ml #146) | nothing |
| [03](03-gkd-arming.md) | GKD rung: arm spec + registration (launch = owner) | P2 | ARMED (ml #147) — launch on owner word | owner ordering |
| [04](04-g2b-watch.md) | G2b verdict watch + cross-recording | P2 | WATCHING — no final_eval yet as of 2026-09-03; rung census: 8.26/7.56/4.83/5.29/5.08/4.57/5.09/5.81/5.78/73.95 | other agents' run |
| [05](05-project-closeouts.md) | Issue closures #38/#41 + legacy alert dismissals | P3 | COMPLETE — #38/#41 closed with cross-refs; all 56 Dependabot alerts auto-FIXED by #48+#78 on rescan (no dismissals needed) | nothing |

## Working rules (unchanged)

- Protocol-matched numbers only; every paper number must trace to
rababa/docs/RESULTS.md (the ledger is ground truth).
- Never fabricate; if a draft cites a superseded number, replace it
with the ledger value — never smooth over.
- Owner-only acts (GKD launch, releases, alert dismissal if API
denies) are prepared, not executed.