Datawhale's Happy-LLM Teaches Anyone to Build an LLM From Scratch
Datawhale's Happy-LLM is a free, structured Chinese-language curriculum that walks you through building a real LLaMA2-style model from scratch — and it just crossed 28,000 GitHub stars.
Takeaways
- Free, structured LLM curriculum: Happy-LLM by Datawhale teaches you to build a LLaMA2-style model from scratch across 7 chapters.
- 28,000+ GitHub stars: One of the most-starred LLM education repos in the Chinese-speaking world, with notable international recognition.
- Ships real model weights: A 215M-parameter base and SFT model are available on ModelScope for download and interactive demo.
- Covers the full training stack: Pre-training, SFT, LoRA/QLoRA fine-tuning, RAG, and Agents — all with working PyTorch code.
- Completely free: CC BY-NC-SA 4.0 license; PDF, PPT slides, and online reading all available at no cost.
- Active community: Extra Chapter blog accepts community PRs; CCF partnership provides free compute for registered learners. GitHub
Happy-LLM is a free, open-source curriculum from Datawhale, China's leading AI open-source community, that teaches you how to build a large language model from the ground up. With over 28,000 GitHub stars and 2,600 forks, it has become one of the most-starred LLM education projects in the Chinese-speaking world , and its English README makes it increasingly accessible beyond that.
The project emerged as a response to the growing demand for deeper understanding of LLM principles beyond surface-level usage. After the success of Datawhale's earlier self-llm project (an open-source LLM usage guide), the community expressed a desire to delve into the underlying architecture and training processes. Happy-LLM is the answer to that demand.
A curriculum, not just a repo
The tutorial is organized into two major parts , Foundations and Practice , spanning seven core chapters plus a community-driven Extra Chapter. The split is deliberate: you build mental models before you touch code.
- Chapters 1-4 (Foundations): NLP basics, the Transformer architecture and attention mechanism, pre-trained language model families (Encoder-only, Encoder-Decoder, Decoder-only), and what makes LLMs different , including emergent abilities and training strategies.
- Chapter 5 (Build it): Implement a LLaMA2-style model in raw PyTorch, train a tokenizer from scratch, and run a full pre-training loop on a small model.
- Chapter 6 (Train it): Use the HuggingFace Transformers ecosystem, DeepSpeed for distributed training, supervised fine-tuning (SFT), and parameter-efficient methods like LoRA and QLoRA.