Skip to main content
Back

Datawhale's Happy-LLM Teaches Anyone to Build an LLM From Scratch

Datawhale's Happy-LLM is a free, structured Chinese-language curriculum that walks you through building a real LLaMA2-style model from scratch — and it just crossed 28,000 GitHub stars.

Repo, Full Access
  • Free, structured LLM curriculum: Happy-LLM by Datawhale teaches you to build a LLaMA2-style model from scratch across 7 chapters.
  • 28,000+ GitHub stars: One of the most-starred LLM education repos in the Chinese-speaking world, with notable international recognition.
  • Ships real model weights: A 215M-parameter base and SFT model are available on ModelScope for download and interactive demo.
  • Covers the full training stack: Pre-training, SFT, LoRA/QLoRA fine-tuning, RAG, and Agents — all with working PyTorch code.
  • Completely free: CC BY-NC-SA 4.0 license; PDF, PPT slides, and online reading all available at no cost.
  • Active community: Extra Chapter blog accepts community PRs; CCF partnership provides free compute for registered learners. GitHub

Happy-LLM is a free, open-source curriculum from Datawhale, China's leading AI open-source community, that teaches you how to build a large language model from the ground up. With over 28,000 GitHub stars and 2,600 forks, it has become one of the most-starred LLM education projects in the Chinese-speaking world , and its English README makes it increasingly accessible beyond that.

The project emerged as a response to the growing demand for deeper understanding of LLM principles beyond surface-level usage. After the success of Datawhale's earlier self-llm project (an open-source LLM usage guide), the community expressed a desire to delve into the underlying architecture and training processes. Happy-LLM is the answer to that demand.

A curriculum, not just a repo

The tutorial is organized into two major parts , Foundations and Practice , spanning seven core chapters plus a community-driven Extra Chapter. The split is deliberate: you build mental models before you touch code.

  • Chapters 1-4 (Foundations): NLP basics, the Transformer architecture and attention mechanism, pre-trained language model families (Encoder-only, Encoder-Decoder, Decoder-only), and what makes LLMs different , including emergent abilities and training strategies.
  • Chapter 5 (Build it): Implement a LLaMA2-style model in raw PyTorch, train a tokenizer from scratch, and run a full pre-training loop on a small model.
  • Chapter 6 (Train it): Use the HuggingFace Transformers ecosystem, DeepSpeed for distributed training, supervised fine-tuning (SFT), and parameter-efficient methods like LoRA and QLoRA.