Tencent's Hy3 Quietly Ships a 295B Open MoE Rivaling Claude
Tencent's rebuilt Hunyuan pipeline delivers a 295B MoE with 21B active params, 256K context, and competitive coding agent scores.
Takeaways
- Tencent open-sourced Hy3 preview, a 295B MoE with 21B active params and 256K context.
- Architecture uses 192 experts with top-8 routing, plus a 3.8B multi-token prediction layer for speculative decoding.
- Scores 74.4% on SWE-bench Verified, 54.4% on Terminal-Bench 2.0, 87.2 on GPQA-Diamond.
- Ships with three reasoning modes (no_think, low, high) via a single runtime flag.
- Deployable via vLLM or SGLang on 8xH200 or 8xH20-3e class hardware.
- First model from Tencent's rebuilt Hunyuan infrastructure, already powering Yuanbao and CodeBuddy.
Tencent's Hunyuan team has quietly shipped what looks like its most serious open-weight release to date. Hy3 preview is a 295B-parameter Mixture-of-Experts language model with only 21B active parameters per token, a 256K context window, and a design that leans hard into coding agents and long-context reasoning rather than chasing academic leaderboards.
The weights are already public on Hugging Face, ModelScope, and GitCode, and the model card is unusually candid: Hy3 preview is the first model trained on Tencent's rebuilt infrastructure, with significant improvements on complex reasoning, instruction following, in-context learning, coding, and agent tasks.
A sparse giant with a speculative-decoding twist
The architecture is where things get interesting. Hy3 preview is a sparse MoE with a 3.8B multi-token prediction (MTP) layer, 192 experts, and top-8 routing. That MTP layer isn't cosmetic. It lets the model draft several tokens ahead in a single forward pass, which pairs naturally with speculative decoding in vLLM and SGLang.
The rest of the spec sheet:
- 80 transformer layers plus 1 MTP layer
- GQA attention with 64 heads and 8 KV heads (head dim 128)
- Hidden size 4096, intermediate size 13312
- Vocabulary of 120,832 tokens
- BF16 precision, 256K context length
One detail worth flagging for people who care about efficiency: unlike traditional MoE where all experts are the same size, Hunyuan introduces a differentiated expert size design, routing tokens of varying difficulty to experts with different capacities, and reports that this significantly outperforms uniform-expert MoE under the same computational budget. They also apply a P-Penalty Loss, which penalizes the model's tendency to favor large experts and encourages it to activate more small ones. Whether this generalizes is an open question, but it's a concrete departure from the uniform-expert dogma that DeepSeek and Mixtral popularized.