depth-pruning

Here is 1 public repository matching this topic...

Prune Qwen2.5-0.5B from 24 to 7 layers, distill it back on 800M tokens, and test what the loss curve can and can't tell you.

pytorch knowledge-distillation model-compression scaling-laws llm fineweb qwen2-5 depth-pruning minitron

To associate your repository with the depth-pruning topic, visit your repo's landing page and select "manage topics."