minimind-3 Lets Anyone Train a Real LLM for $0.42 on One GPU
MiniMind lets you train a full 64M-parameter LLM from scratch — including RL, tool use, and agentic training — on a single GPU for under $0.50.
Takeaways
- MiniMind trains a 64M-parameter LLM from scratch on a single RTX 3090 in ~2 hours for ~$0.42.
- The latest release, minimind-3, aligns architecture with Qwen3/Qwen3-MoE and adds native Agentic RL training.
- Full pipeline covered: Pretrain → SFT → LoRA → DPO → PPO/GRPO/CISPO → multi-turn Agentic RL with tool use.
- All algorithms implemented in native PyTorch with no third-party training library abstractions — every line is readable.
- Compatible with vLLM, ollama, llama.cpp, SGLang, and OpenAI-compatible API for deployment; models on HuggingFace.
- Not a production model — benchmarks hover near random chance — but cited in ICML 2025 and used in multiple research papers.
MiniMind is an open-source project that does something most ML practitioners assume requires a data center: it lets you train a working large language model completely from scratch, on a single consumer GPU, in about two hours, for roughly $0.40. With over 50,000 GitHub stars, it has become one of the most-starred LLM education repositories in existence , and the latest release, minimind-3, is the most complete version yet.
What problem does this actually solve?
Most people who want to "understand" LLMs end up doing one of two things: calling an API, or running a LoRA fine-tune on top of an existing model using a high-level library like transformers or trl. Both approaches hide the internals behind thick abstractions. MiniMind takes the opposite stance: pure PyTorch, no black boxes, understand every line of code. The goal is not to produce a state-of-the-art model, but to give you a fully transparent, reproducible training pipeline you can actually read and modify.
The project's author frames it well: using a framework to fine-tune an existing model is like teaching Newton to use a smartphone , interesting, but it misses the point. MiniMind is about building the physics from first principles.
What's in the box
The complete pipeline covers: Tokenizer → Pretrain → SFT → LoRA → DPO → PPO/GRPO/CISPO → Agentic RL, plus YaRN length extrapolation. Every stage is implemented natively in PyTorch with no dependency on third-party training wrappers. Here's what that looks like end-to-end:
- Pretraining , next-token prediction on a cleaned Chinese/English corpus to build base language knowledge
- Supervised Fine-Tuning (SFT) , instruction-following using multi-turn conversation data, with tool-call samples already mixed in
- LoRA , low-rank adaptation for domain-specific fine-tuning (medical Q&A, custom personas, etc.), implemented from scratch without
peft - DPO , Direct Preference Optimization for human preference alignment, also implemented natively
- RLAIF (PPO / GRPO / CISPO) , three reinforcement learning algorithms with a unified theoretical framework
- Agentic RL , multi-turn tool-use training with delayed rewards and rollout engine decoupling
- Knowledge Distillation , both black-box (SFT on teacher outputs) and white-box (KL divergence on token distributions)

The flagship model: minimind-3
The latest release, minimind-3, aligns its architecture with the Qwen3/Qwen3-MoE ecosystem: Dense ~64M parameters, MoE ~198M/A64M. The Dense model uses a Transformer Decoder-Only architecture with Pre-Norm + RMSNorm, SwiGLU activations, RoPE positional encoding with YaRN extrapolation support, and Grouped Query Attention (