Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.
moe mixture-of-experts kv-cache llm rmsnorm swiglu pytorch-llm byte-pair-tokenizer rotational-positional-embedding
-
Updated
Nov 18, 2024 - Python