From f7cf34fcde3a7ec445bc2a9bf5b6053ab3351bc9 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 5 Sep 2026 11:59:48 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20GKD=20verdict=20=E2=80=94=206.0036=20GA?= =?UTF-8?q?TE=20FAILED;=20the=20lever=20ladder=20closes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On-policy reverse-KL is the worst rung measured, +1.18pp over the 4.8218 control; bootstrap +3.4083 CI 3.109-3.743; teacher reproduces 2.289. Registered prediction 4.30-4.65 missed. With MTP, register swap and add, the frontier-training toolkit is 0-for-4 on byte-level student distillation; optimizer, fresher labels, epochs remain the only positives. run-012 absorbed nine preemptions via checkpoint-resume. PUBLICATION-NOTES: lever table row, negatives exhibit extended to the four-way set, framing line closed. --- docs/EXPERIMENTS.md | 50 +++++++++++++++++++++------------------ docs/PUBLICATION-NOTES.md | 21 ++++++++++------ 2 files changed, 41 insertions(+), 30 deletions(-) diff --git a/docs/EXPERIMENTS.md b/docs/EXPERIMENTS.md index 6e34ef9..0baa7e0 100644 --- a/docs/EXPERIMENTS.md +++ b/docs/EXPERIMENTS.md @@ -318,30 +318,34 @@ All rows passed the CER parity gate at release. Readings: (−0.47pp), and epochs (−0.25pp). What remains is GKD (on-policy exposure, run-012 in flight) — after which the ladder is closed. -## GKD — on-policy distillation rung (REGISTERED/ARMED 2026-09-03) +## GKD — on-policy distillation rung (COMPLETE 2026-09-05) -- **Status:** REGISTERED, launch pending owner ordering (the last - lever with no data; TODO.substantiate/03). Implementation follows - this registration verbatim when launched. -- **Hypothesis:** the client tier's remaining ~2.0pp over the teacher - is domain-shaped exposure: the student never trains on its own - decode distribution. On-policy GKD (sequences sampled from the - student during training, scored against the frozen r7 teacher's - targets/logits) attacks exactly that. -- **Design:** spec `ara-diac-small-2-gkd`, control = run-006 verbatim - (30k units, identical steps/schedule/optimizer); delta = GKD loss - mixing — student-sampled sequences (temperature-matched to greedy - inference) blended into the sequence-KD objective at a fixed ratio, - annealed to zero over the final third. -- **Pre-agreed gate:** adopt at <= 4.5218 full-set windowed DER; - honest-report band [4.5218, 4.8218); investigate if worse. -- **Prediction (registered):** 4.30-4.65 — the E6 swap-negative says - register mix at constant budget is not the lever; on-policy - exposure is the remaining untested reading of the domain - attribution. Note: run-006's canonical control is 4.8218 (E4-era - eval); a later auto-chain re-eval printed 5.0821 for the same run — - the ledger's 4.8218 stays canonical; a GKD verdict compares within - one eval pass. +- **Status:** COMPLETE — **GATE FAILED: 6.0036** full-set windowed DER + (n=1200; teacher reproduces 2.289; paired bootstrap student−teacher + +3.4083, CI [3.109, 3.743]). NOT ADOPTED. The registered prediction + (4.30-4.65) missed badly; honest-report band also breached — this + is the worst rung measured, +1.18pp over the 4.8218 control. Run + run-012-r7-muon-gkd: 10,995 steps, final CE 0.0076, nine server + preemptions absorbed by checkpoint-resume (no measured work lost); + labels sha256 e70ce991d15a8c810b83e2b5401f1410293844c623ffefa646b + 19fb94a6180df. +- **Hypothesis (registered):** the client tier's remaining ~2.0pp over + the teacher is domain-shaped exposure: the student never trains on + its own decode distribution. On-policy GKD attacks exactly that. +- **Design (as run):** control run-006 verbatim (30k units, identical + steps/schedule/optimizer); delta = reverse-KL on student-sampled + sequences (every 4th step, sub-batch 2, temp 1.0, cap 1024) scored + by the frozen r7 teacher, ratio 0.3 annealed to zero over the final + third (ml #148; run id renamed run-012 in #151). +- **Read:** the domain-exposure reading of the residual joins the + register readings as a THIRD negative (E6 swap 5.8057, G2b add + 4.8231, GKD on-policy 6.0036) — the program's final datum. The + ladder closes: optimizer (−2.96pp), fresher teacher labels + (−0.47pp), and epochs (−0.25pp) are the only measured positives; + MTP-aux, register diversification (both directions), on-policy + distillation, and depth-cut all regressed or went flat. At SFT + convergence on clean supervision, byte-level student distillation + tolerates nothing but optimization and supervision quality. ## Parked diff --git a/docs/PUBLICATION-NOTES.md b/docs/PUBLICATION-NOTES.md index 713dd40..d1111ac 100644 --- a/docs/PUBLICATION-NOTES.md +++ b/docs/PUBLICATION-NOTES.md @@ -81,13 +81,19 @@ failing in the swap direction; G2b label-scale ADD (2026-09-04, verdict recorded per TODO.substantiate 04): 4.8231 — 48k total (+18k classical) at 6 epochs is FLAT vs the 3ep control and 0.25pp worse than news-only G2a (4.5701): the add neutralized the epoch -gain, closing the domain hypothesis negative in BOTH directions.** -The E5/E6/G2b set is the paper's data-vs-architecture exhibit: -two levers from the frontier-LLM literature (MTP, register -diversification — swap or add) plus the label-scale hypothesis all -regressed or flat on byte-level student distillation — the levers -that moved the rung were optimizer -(E3), fresher teacher labels (E4), and epochs (G2a). +gain, closing the domain hypothesis negative in BOTH directions; GKD +on-policy distillation (2026-09-05, the program's final rung, +run-012): 6.0036 — reverse-KL on student-sampled sequences scored by +the frozen teacher is the WORST rung measured (+1.18pp over control), +the domain-exposure reading of the residual joining the register +readings as the third negative.** +The E5/E6/G2b/GKD set is the paper's data-vs-architecture exhibit: +three levers from the frontier-LLM literature (MTP, register +diversification — swap or add, on-policy GKD) plus the label-scale +hypothesis all regressed or flat on byte-level student distillation — +the levers that moved the rung were optimizer +(E3), fresher teacher labels (E4), and epochs (G2a), and nothing +else. The program's measured ladder is closed. ### 9. Muon optimizer A/B on the memory student (E3) — LANDED 2026-08-28 **4.8287 vs 7.5553 full-set (−2.727pp from the optimizer alone); adopt @@ -234,6 +240,7 @@ The decomposition for paper B, every line full-set with brackets: | register swap (E6, 3ep) | 5.8057 | negative | | register add (G2b, 6ep) | 4.8231 | delta 2.37 [2.19, 2.55] | | MTP-aux (E5, 3ep) | 5.0853 | negative (confound disclosed) | +| on-policy GKD (run-012) | 6.0036 | delta 3.41 [3.11, 3.74] — worst rung | | depth halved (lite, 6ep) | 5.784 | delta 3.25 [3.03, 3.49] | Paper-B framing: levers compose roughly additively (optimizer >>