MiniMind-3 Trains a Real AI Model From Scratch for $0.42
MiniMind-3 lets you train a working 64M-parameter LLM from scratch in 2 hours on a single GPU, with full RL, tool calling, and agentic training included
Takeaways
- MiniMind-3 released: Full architectural overhaul aligned to Qwen3, with a 64M Dense and 198M/A64M MoE model trained from scratch on a single RTX 3090.
- 2 hours, ~$0.42: Complete pretrain + SFT pipeline runs in ~2.3 hours on one GPU, costing under half a dollar in cloud compute.
- Full RL stack included: Native PyTorch implementations of DPO, PPO, GRPO, and CISPO — plus a new Agentic RL script with multi-turn tool-use training.
- Tool calling baked in: ~100K synthetic tool-call samples mixed into default SFT data; adaptive thinking toggled at inference via
open_thinkingflag. - Ecosystem compatible: Works with vLLM, ollama, llama.cpp, SGLang, and Transformers; OpenAI-compatible API server included.
- 54.8K GitHub stars: Cited in ICML 2025 and a Tsinghua University Press book; weights on HuggingFace and via
ollama run jingyaogong/minimind-3.
MiniMind is an open-source project with a simple, almost absurd premise: train a real, conversational language model completely from scratch, on a single consumer GPU, in about two hours, for roughly $0.40. No pretrained weights to fine-tune, no black-box abstractions, no cloud bill. Just PyTorch, your GPU, and every line of code written by hand. The project has now crossed 54,000 GitHub stars and just shipped its biggest update yet with MiniMind-3, a full architectural overhaul that brings the model in line with Qwen3, adds native agentic reinforcement learning, tool calling, and adaptive thinking , all still running on a single RTX 3090.
What just shipped
The MiniMind-3 release is a ground-up rebuild. The current flagship model is a 64M Dense model and a 198M/A64M MoE variant, both aligned to the Qwen3/Qwen3-MoE architecture. The tokenizer was updated to BPE + ByteLevel with dedicated tool-call and thinking tokens. The training pipeline now covers the full modern LLM stack:
- Pretrain on text-to-text data
- SFT (supervised fine-tuning) with tool call samples already baked in
- LoRA fine-tuning, implemented from scratch without the
peftlibrary - DPO for preference alignment
- RLAIF via PPO, GRPO, and CISPO , all native PyTorch
- Agentic RL with multi-turn tool use via a new
train_agent.pyscript
The project is compatible with vLLM, ollama, llama.cpp, SGLang, and the Transformers ecosystem, and exposes an OpenAI-compatible API endpoint so you can plug it straight into FastGPT, Open-WebUI, or Dify.

The actual cost of training from zero
The model can be trained on a single 3090 GPU in just 2 hours. That 2-hour figure refers specifically to the SFT stage running 1 epoch on the mini dataset. Here is what the full breakdown looks like on a single 3090, with GPU rental at roughly ¥1.3/hour (~$0.18/hour):
| Model | Pretrain (mini) | SFT (mini) | Total time | Total cost |
|---|---|---|---|---|
| MiniMind-3 (64M Dense) | ≈1.21h | ≈1.10h | ≈2.31h | ≈¥3.0 (~$0.42) |
| MiniMind-3-MoE (198M/A64M) | ≈1.69h | ≈1.54h | ≈3.23h | ≈¥4.2 (~$0.58) |
The "mini" datasets (pretrain_t2t_mini.jsonl at 1.2GB and sft_t2t_mini.jsonl at 1.6GB) are designed for fast reproduction. The full datasets are 10GB and 14GB respectively, for anyone who wants to push quality further. All datasets are freely available on HuggingFace and ModelScope.
Architecture: every design choice is visible
MiniMind-3's Dense model is a standard Transformer Decoder-Only architecture, but the implementation philosophy is what sets it apart. The project provides an excellent platform for AI learners and researchers to understand the internal mechanisms of LLMs. Every component , attention, RoPE, MoE routing, GRPO loss, LoRA adapters , is written in raw PyTorch with no hidden abstractions.