Skip to main content
Back

Datawhale Rebuilds Stanford's CS336 Into a Chinese Open-Source LLM Course

A Chinese-language adaptation of Stanford's CS336 bundles 16 chapters and 6 hands-on assignments covering the full LLM stack, from BPE to GRPO.

Repo, Full Access
  • Datawhale released diy-llm, a Chinese adaptation of Stanford CS336 with 16 chapters and 6 coding assignments.
  • The repo passed 1.7k stars and 180 forks, trending on GitHub this week.
  • Assignments walk from hand-written BPE tokenizers to FlashAttention-2 in Triton to GRPO alignment.
  • Systems chapters cover FSDP, ZeRO, PagedAttention, speculative decoding, GPTQ and AWQ quantization.
  • Localization adds Qwen and DeepSeek case studies plus a downloadable PDF of lecture notes.
  • Licensed CC BY-NC-SA 4.0, with full source at datawhalechina.github.io/diy-llm.

Stanford’s CS336, Language Modeling from Scratch, teaches students to build the components behind a language model. Datawhale’s open-source project repository adapts that material for Chinese learners through localized notes, examples based on Qwen and DeepSeek, and six coding assignments that cover the model pipeline from tokenization to alignment.

The repository has attracted more than 1,700 GitHub stars and appeared in GitHub Trending. Its code-heavy structure gives developers a guided route through model architecture, training systems, data preparation, scaling, alignment, and evaluation.

From tokenizer to GRPO in six assignments

Across 16 chapters, each assignment builds on infrastructure introduced earlier in the course. Learners finish with a readable codebase that touches every major stage of language-model development.

Assignment What developers build Core concepts
1. Basics Write a BPE tokenizer, model architecture, and optimizer, then train a minimal language model. Tokenization, transformer components, optimization, and training loops
2. Systems Profile training, implement FlashAttention-2 in Triton, and add distributed training. GPU performance, custom kernels, memory use, and parallelism
3. Scaling Fit scaling laws and use them to predict model behavior at larger compute budgets. Compute allocation, empirical prediction, and experiment design
4. Data Convert raw Common Crawl data into a filtered and deduplicated pretraining dataset. Web-data processing, quality filtering, and duplicate removal
5. Alignment Apply supervised fine-tuning and GRPO-style reinforcement learning to mathematical reasoning tasks. Post-training, reward design, and rule-based verification
6. Evaluation Evaluate models across multiple dimensions with lm-evaluation-harness and EvalScope. Benchmark execution, metrics, and reproducible comparisons

Technical terms in context

Byte-pair encoding
BPE converts text into reusable token units that a model can process.
FlashAttention-2 and Triton
FlashAttention-2 reduces the memory traffic required by attention. Triton provides a Python-based language for writing the corresponding GPU kernels.
Scaling laws
Scaling laws estimate how model performance changes with parameter count, data volume, and training compute.
SFT and GRPO
Supervised fine-tuning trains on curated examples. Group Relative Policy Optimization uses relative rewards among generated answers to improve a model’s policy.

The wider technical syllabus

  • Model internals: rotary position embeddings, known as RoPE; RMSNorm; SwiGLU; AdamW; Pre-Norm and Post-Norm; and mixture-of-experts routing with load balancing.
  • Training systems: GPU memory bandwidth, kernel fusion, Tensor Cores, Triton, data parallelism, model parallelism, pipeline parallelism, ZeRO stages 1 through 3, and Fully Sharded Data Parallel.
  • Scaling and inference: Chinchilla-style compute-optimal training, key-value caches, speculative decoding, PagedAttention, and GPTQ and AWQ quantization.
  • Alignment: supervised fine-tuning, Direct Preference Optimization, Proximal Policy Optimization, and Group Relative Policy Optimization with rule-based verifiers similar to those used for DeepSeek-R1.

Chinese models anchor the curriculum

diy-llm reorganizes CS336’s public materials for Chinese learners, adds prerequisite mathematics and deep-learning explanations, and uses Qwen and DeepSeek as recurring case studies alongside established Western model families. The result aligns the theory with models and tooling that many Chinese developers already encounter.

Local operating constraints also shape the project. A downloadable PDF supports offline study, and community patches add Apple Metal Performance Shaders support to the Assignment 1 training script. That MPS path allows small-scale debugging on a MacBook before moving GPU-intensive work to cloud or CUDA hardware.

GPU costs and unfinished chapters set the limits

Full pretraining and reinforcement-learning runs require substantial GPU capacity. Learners without local hardware or cloud credits can still exercise reduced configurations, inspect the implementation, and test individual components, though those runs cannot reproduce the behavior or performance of large training jobs.

Several sections remained incomplete at the time of writing:

  • Chapter 1’s experiment-tracking material was marked as a work in progress.
  • Chapter 16’s extended coverage of inference and LeCun-style future model directions was still being updated.
  • The English translation lagged behind the Chinese edition.

Best suited to experienced PyTorch users

Developers who already use model APIs or fine-tuning libraries can use the course to examine the layers beneath those abstractions. The expected foundation includes practical PyTorch experience, working knowledge of linear algebra and probability, and enough time to implement and debug training code.

Product-focused topics such as prompt engineering, agents, and retrieval-augmented generation receive limited direct coverage. The curriculum concentrates on model internals, training infrastructure, pretraining data, alignment methods, and evaluation.

Assignment 1 provides the lowest-cost entry point because its tokenizer, model, optimizer, and small training run can be developed incrementally. Later systems and alignment assignments require advance planning for compatible GPUs, cloud spending, and longer experiment cycles.