Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 27 additions & 23 deletions docs/EXPERIMENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -318,30 +318,34 @@ All rows passed the CER parity gate at release. Readings:
(−0.47pp), and epochs (−0.25pp). What remains is GKD (on-policy
exposure, run-012 in flight) — after which the ladder is closed.

## GKD — on-policy distillation rung (REGISTERED/ARMED 2026-09-03)
## GKD — on-policy distillation rung (COMPLETE 2026-09-05)

- **Status:** REGISTERED, launch pending owner ordering (the last
lever with no data; TODO.substantiate/03). Implementation follows
this registration verbatim when launched.
- **Hypothesis:** the client tier's remaining ~2.0pp over the teacher
is domain-shaped exposure: the student never trains on its own
decode distribution. On-policy GKD (sequences sampled from the
student during training, scored against the frozen r7 teacher's
targets/logits) attacks exactly that.
- **Design:** spec `ara-diac-small-2-gkd`, control = run-006 verbatim
(30k units, identical steps/schedule/optimizer); delta = GKD loss
mixing — student-sampled sequences (temperature-matched to greedy
inference) blended into the sequence-KD objective at a fixed ratio,
annealed to zero over the final third.
- **Pre-agreed gate:** adopt at <= 4.5218 full-set windowed DER;
honest-report band [4.5218, 4.8218); investigate if worse.
- **Prediction (registered):** 4.30-4.65 — the E6 swap-negative says
register mix at constant budget is not the lever; on-policy
exposure is the remaining untested reading of the domain
attribution. Note: run-006's canonical control is 4.8218 (E4-era
eval); a later auto-chain re-eval printed 5.0821 for the same run —
the ledger's 4.8218 stays canonical; a GKD verdict compares within
one eval pass.
- **Status:** COMPLETE — **GATE FAILED: 6.0036** full-set windowed DER
(n=1200; teacher reproduces 2.289; paired bootstrap student−teacher
+3.4083, CI [3.109, 3.743]). NOT ADOPTED. The registered prediction
(4.30-4.65) missed badly; honest-report band also breached — this
is the worst rung measured, +1.18pp over the 4.8218 control. Run
run-012-r7-muon-gkd: 10,995 steps, final CE 0.0076, nine server
preemptions absorbed by checkpoint-resume (no measured work lost);
labels sha256 e70ce991d15a8c810b83e2b5401f1410293844c623ffefa646b
19fb94a6180df.
- **Hypothesis (registered):** the client tier's remaining ~2.0pp over
the teacher is domain-shaped exposure: the student never trains on
its own decode distribution. On-policy GKD attacks exactly that.
- **Design (as run):** control run-006 verbatim (30k units, identical
steps/schedule/optimizer); delta = reverse-KL on student-sampled
sequences (every 4th step, sub-batch 2, temp 1.0, cap 1024) scored
by the frozen r7 teacher, ratio 0.3 annealed to zero over the final
third (ml #148; run id renamed run-012 in #151).
- **Read:** the domain-exposure reading of the residual joins the
register readings as a THIRD negative (E6 swap 5.8057, G2b add
4.8231, GKD on-policy 6.0036) — the program's final datum. The
ladder closes: optimizer (−2.96pp), fresher teacher labels
(−0.47pp), and epochs (−0.25pp) are the only measured positives;
MTP-aux, register diversification (both directions), on-policy
distillation, and depth-cut all regressed or went flat. At SFT
convergence on clean supervision, byte-level student distillation
tolerates nothing but optimization and supervision quality.

## Parked

Expand Down
21 changes: 14 additions & 7 deletions docs/PUBLICATION-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,13 +81,19 @@ failing in the swap direction; G2b label-scale ADD (2026-09-04,
verdict recorded per TODO.substantiate 04): 4.8231 — 48k total
(+18k classical) at 6 epochs is FLAT vs the 3ep control and 0.25pp
worse than news-only G2a (4.5701): the add neutralized the epoch
gain, closing the domain hypothesis negative in BOTH directions.**
The E5/E6/G2b set is the paper's data-vs-architecture exhibit:
two levers from the frontier-LLM literature (MTP, register
diversification — swap or add) plus the label-scale hypothesis all
regressed or flat on byte-level student distillation — the levers
that moved the rung were optimizer
(E3), fresher teacher labels (E4), and epochs (G2a).
gain, closing the domain hypothesis negative in BOTH directions; GKD
on-policy distillation (2026-09-05, the program's final rung,
run-012): 6.0036 — reverse-KL on student-sampled sequences scored by
the frozen teacher is the WORST rung measured (+1.18pp over control),
the domain-exposure reading of the residual joining the register
readings as the third negative.**
The E5/E6/G2b/GKD set is the paper's data-vs-architecture exhibit:
three levers from the frontier-LLM literature (MTP, register
diversification — swap or add, on-policy GKD) plus the label-scale
hypothesis all regressed or flat on byte-level student distillation —
the levers that moved the rung were optimizer
(E3), fresher teacher labels (E4), and epochs (G2a), and nothing
else. The program's measured ladder is closed.

### 9. Muon optimizer A/B on the memory student (E3) — LANDED 2026-08-28
**4.8287 vs 7.5553 full-set (−2.727pp from the optimizer alone); adopt
Expand Down Expand Up @@ -234,6 +240,7 @@ The decomposition for paper B, every line full-set with brackets:
| register swap (E6, 3ep) | 5.8057 | negative |
| register add (G2b, 6ep) | 4.8231 | delta 2.37 [2.19, 2.55] |
| MTP-aux (E5, 3ep) | 5.0853 | negative (confound disclosed) |
| on-policy GKD (run-012) | 6.0036 | delta 3.41 [3.11, 3.74] — worst rung |
| depth halved (lite, 6ep) | 5.784 | delta 3.25 [3.03, 3.49] |

Paper-B framing: levers compose roughly additively (optimizer >>
Expand Down
Loading