Datawhale Rebuilds Stanford's CS336 Into a Chinese Open-Source LLM Course
A Chinese-language adaptation of Stanford's CS336 bundles 16 chapters and 6 hands-on assignments covering the full LLM stack, from BPE to GRPO.
Takeaways
- Datawhale released diy-llm, a Chinese adaptation of Stanford CS336 with 16 chapters and 6 coding assignments.
- The repo passed 1.7k stars and 180 forks, trending on GitHub this week.
- Assignments walk from hand-written BPE tokenizers to FlashAttention-2 in Triton to GRPO alignment.
- Systems chapters cover FSDP, ZeRO, PagedAttention, speculative decoding, GPTQ and AWQ quantization.
- Localization adds Qwen and DeepSeek case studies plus a downloadable PDF of lecture notes.
- Licensed CC BY-NC-SA 4.0, with full source at datawhalechina.github.io/diy-llm.
Stanford’s CS336, Language Modeling from Scratch, teaches students to build the components behind a language model. Datawhale’s open-source project repository adapts that material for Chinese learners through localized notes, examples based on Qwen and DeepSeek, and six coding assignments that cover the model pipeline from tokenization to alignment.
The repository has attracted more than 1,700 GitHub stars and appeared in GitHub Trending. Its code-heavy structure gives developers a guided route through model architecture, training systems, data preparation, scaling, alignment, and evaluation.
From tokenizer to GRPO in six assignments
Across 16 chapters, each assignment builds on infrastructure introduced earlier in the course. Learners finish with a readable codebase that touches every major stage of language-model development.
| Assignment | What developers build | Core concepts |
|---|---|---|
| 1. Basics | Write a BPE tokenizer, model architecture, and optimizer, then train a minimal language model. | Tokenization, transformer components, optimization, and training loops |
| 2. Systems | Profile training, implement FlashAttention-2 in Triton, and add distributed training. | GPU performance, custom kernels, memory use, and parallelism |
| 3. Scaling | Fit scaling laws and use them to predict model behavior at larger compute budgets. | Compute allocation, empirical prediction, and experiment design |
| 4. Data | Convert raw Common Crawl data into a filtered and deduplicated pretraining dataset. | Web-data processing, quality filtering, and duplicate removal |
| 5. Alignment | Apply supervised fine-tuning and GRPO-style reinforcement learning to mathematical reasoning tasks. | Post-training, reward design, and rule-based verification |
| 6. Evaluation | Evaluate models across multiple dimensions with lm-evaluation-harness and EvalScope. | Benchmark execution, metrics, and reproducible comparisons |
Technical terms in context
- Byte-pair encoding
- BPE converts text into reusable token units that a model can process.
- FlashAttention-2 and Triton
- FlashAttention-2 reduces the memory traffic required by attention. Triton provides a Python-based language for writing the corresponding GPU kernels.
- Scaling laws
- Scaling laws estimate how model performance changes with parameter count, data volume, and training compute.
- SFT and GRPO
- Supervised fine-tuning trains on curated examples. Group Relative Policy Optimization uses relative rewards among generated answers to improve a model’s policy.
The wider technical syllabus
- Model internals: rotary position embeddings, known as RoPE; RMSNorm; SwiGLU; AdamW; Pre-Norm and Post-Norm; and mixture-of-experts routing with load balancing.
- Training systems: GPU memory bandwidth, kernel fusion, Tensor Cores, Triton, data parallelism, model parallelism, pipeline parallelism, ZeRO stages 1 through 3, and Fully Sharded Data Parallel.
- Scaling and inference: Chinchilla-style compute-optimal training, key-value caches, speculative decoding, PagedAttention, and GPTQ and AWQ quantization.
- Alignment: supervised fine-tuning, Direct Preference Optimization, Proximal Policy Optimization, and Group Relative Policy Optimization with rule-based verifiers similar to those used for DeepSeek-R1.
Chinese models anchor the curriculum
diy-llm reorganizes CS336’s public materials for Chinese learners, adds prerequisite mathematics and deep-learning explanations, and uses Qwen and DeepSeek as recurring case studies alongside established Western model families. The result aligns the theory with models and tooling that many Chinese developers already encounter.
Local operating constraints also shape the project. A downloadable PDF supports offline study, and community patches add Apple Metal Performance Shaders support to the Assignment 1 training script. That MPS path allows small-scale debugging on a MacBook before moving GPU-intensive work to cloud or CUDA hardware.
GPU costs and unfinished chapters set the limits
Full pretraining and reinforcement-learning runs require substantial GPU capacity. Learners without local hardware or cloud credits can still exercise reduced configurations, inspect the implementation, and test individual components, though those runs cannot reproduce the behavior or performance of large training jobs.
Several sections remained incomplete at the time of writing:
- Chapter 1’s experiment-tracking material was marked as a work in progress.
- Chapter 16’s extended coverage of inference and LeCun-style future model directions was still being updated.
- The English translation lagged behind the Chinese edition.
Best suited to experienced PyTorch users
Developers who already use model APIs or fine-tuning libraries can use the course to examine the layers beneath those abstractions. The expected foundation includes practical PyTorch experience, working knowledge of linear algebra and probability, and enough time to implement and debug training code.
Product-focused topics such as prompt engineering, agents, and retrieval-augmented generation receive limited direct coverage. The curriculum concentrates on model internals, training infrastructure, pretraining data, alignment methods, and evaluation.
Assignment 1 provides the lowest-cost entry point because its tokenizer, model, optimizer, and small training run can be developed incrementally. Later systems and alignment assignments require advance planning for compatible GPUs, cloud spending, and longer experiment cycles.