Skip to content

docs: Hebrew layerdrop verdict — depth-cut collapse #179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Sep 5, 2026
Merged

docs: Hebrew layerdrop verdict — depth-cut collapse #179

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions docs/RESULTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -521,3 +521,30 @@ levers: on-policy distillation — GKD in flight; label-distribution
effects). G2a (2.1) remains the best student; G2b stands as the
closing negative of the data-side program. Provenance: labels sha256
b59e2f56 (235.0MB).

## heb-diac-small-s46-layerdrop — the depth-cut does NOT transfer: 77.48 DER (2026-09-05)

Item 04 (TODO.publish-client): the Arabic width/depth finding
replicated on Hebrew — encoder 12->6 verbatim layer copy from
pretrained ByT5-small, single variable vs run-002-s46 (same s46
teacher, hebrew-v4 corpus, 3 epochs, logit-KD recipe). Training
converged normally (val_loss 0.550); the full Nakdimon gate did not:

| Model | DER (n=1864) |
|---|---|
| teacher s46 | 23.72 (reproduces exactly) |
| full-depth student (run-002, 1.1) | 30.38 |
| **layerdrop student (run-003)** | **77.48** — collapse |

Paired delta +53.77pp [51.64, 55.92], p=0.

Verdict: the "depth is the compressible axis" finding does NOT
transfer as-is. Confound, stated honestly: the Arabic rung used
sequence-KD (teacher labels, Muon, 6 epochs); this run used the Hebrew
lineage's logit-KD (alpha-KL + CE) at its native 3 epochs — so the
collapse may be recipe-dependent (depth-cut + logit-KD), not purely
linguistic. Either way the cross-lingual generalization claim is
closed as a negative: depth-compressibility is NOT a universal
property of pretrained ByT5-small; it held under one recipe on one
language. Paper B's depth paragraph is scoped accordingly (this entry
is its counterexample).
Loading