Tencent's Hunyuan-A13B Beats OpenAI o1 on Math With 13B Active Parameters
Tencent open-sourced an 80B-total, 13B-active MoE model with dual thinking modes, 256K context, and benchmark scores rivaling much larger frontier models.
Takeaways
- Tencent open-sourced Hunyuan-A13B-Instruct, an 80B-total, 13B-active MoE model
- Architecture uses 1 shared plus 64 routed experts, 8 activated per token, 32 layers, GQA
- Beats OpenAI o1 on AIME 2024 (87.3 vs 74.3) and leads on agent benchmarks BFCL v3 and C3-Bench
- Dual reasoning modes via /think and /no_think flags, plus native 256K context window
- Ships with FP8, GPTQ Int4, and GGUF quantizations, vLLM, SGLang, TensorRT-LLM support
- Trained on 20T tokens with NTK-aware long-context adaptation, code and report on GitHub
Tencent has released Hunyuan-A13B-Instruct, a sparse Mixture-of-Experts language model with 80 billion total parameters but only 13 billion active per forward pass. The pitch is straightforward: compute costs closer to a 13B dense model, performance closer to a 70B+ one. It also ships with a per-request toggle between fast and slow reasoning modes, which turns out to be genuinely useful for production deployments.
Architecture at a glance
The fine-grained MoE design runs one shared expert alongside 64 non-shared experts, with 8 experts activated per token. Thirty-two layers, SwiGLU activations, a 128K vocabulary, and grouped-query attention round out the stack. GQA handles memory efficiency during long-context inference. Training followed a three-stage curriculum: a 20-trillion-token pretraining phase, fast annealing, and a long-context adaptation stage that stretched the context window first to 32K then to 256K tokens using NTK-aware positional encoding.
Fast thinking and slow thinking in one model
Developers can switch reasoning depth per request using either a chat-template flag or an inline prompt tag. Prepend /think to trigger full chain-of-thought reasoning; prepend /no_think