Fareed Khan's Open-Source Repo Teaches Anyone to Build a 2B LLM
A trending GitHub project walks through every line of PyTorch needed to train a 13M or 2B parameter transformer on a single GPU using The Pile dataset.
Takeaways
- Fareed Khan's train-llm-from-scratch repo hit 4,200+ stars with 2,600 added this week.
- Trains a 13M parameter transformer on a free Colab or Kaggle T4 GPU.
- Scales up to a 2.14 billion parameter model on a single RTX 3090 or A100.
- Uses The Pile dataset and OpenAI's r50k_base tiktoken vocabulary.
- 13M parameters is the empirical floor for coherent grammar and spelling.
- MIT licensed, full PyTorch source plus an SFT and RLHF notebook included.
A new open-source repository is teaching people how to build a working language model from raw PyTorch, without hiding anything behind a high-level library. The project, train-llm-from-scratch by Fareed Khan, has rocketed up the GitHub trending list with over 4,200 stars and roughly 2,600 added in a single week.
The pitch is simple. It implements a transformer model from scratch using PyTorch, based on the paper Attention is All You Need, with scripts to train your own billion or million parameter LLM using a single GPU. The codebase is small, fully MIT licensed, and walks through every component, from tokenization to text generation.
Two models, one codebase
The repo ships configurations for two sizes. The lightweight variant uses a 128-dim embedding, 8 attention heads, and a single transformer block to produce a roughly 13 million parameter model. The flagship config scales to a 512 token context, 2048 embedding dimension, 16 attention heads, and 64 transformer blocks, which clocks in at 2,141,346,251 parameters.