Skip to main content
Back

Tencent's Hy3 Quietly Ships a 295B Open MoE Rivaling Claude

Tencent's rebuilt Hunyuan pipeline delivers a 295B MoE with 21B active params, 256K context, and competitive coding agent scores.

  • Tencent open-sourced Hy3 preview, a 295B MoE with 21B active params and 256K context.
  • Architecture uses 192 experts with top-8 routing, plus a 3.8B multi-token prediction layer for speculative decoding.
  • Scores 74.4% on SWE-bench Verified, 54.4% on Terminal-Bench 2.0, 87.2 on GPQA-Diamond.
  • Ships with three reasoning modes (no_think, low, high) via a single runtime flag.
  • Deployable via vLLM or SGLang on 8xH200 or 8xH20-3e class hardware.
  • First model from Tencent's rebuilt Hunyuan infrastructure, already powering Yuanbao and CodeBuddy.

Tencent's Hunyuan team has quietly shipped what looks like its most serious open-weight release to date. Hy3 preview is a 295B-parameter Mixture-of-Experts language model with only 21B active parameters per token, a 256K context window, and a design that leans hard into coding agents and long-context reasoning rather than chasing academic leaderboards.

The weights are already public on Hugging Face, ModelScope, and GitCode, and the model card is unusually candid: Hy3 preview is the first model trained on Tencent's rebuilt infrastructure, with significant improvements on complex reasoning, instruction following, in-context learning, coding, and agent tasks.

A sparse giant with a speculative-decoding twist

The architecture is where things get interesting. Hy3 preview is a sparse MoE with a 3.8B multi-token prediction (MTP) layer, 192 experts, and top-8 routing. That MTP layer isn't cosmetic. It lets the model draft several tokens ahead in a single forward pass, which pairs naturally with speculative decoding in vLLM and SGLang.

The rest of the spec sheet:

  • 80 transformer layers plus 1 MTP layer
  • GQA attention with 64 heads and 8 KV heads (head dim 128)
  • Hidden size 4096, intermediate size 13312
  • Vocabulary of 120,832 tokens
  • BF16 precision, 256K context length

One detail worth flagging for people who care about efficiency: unlike traditional MoE where all experts are the same size, Hunyuan introduces a differentiated expert size design, routing tokens of varying difficulty to experts with different capacities, and reports that this significantly outperforms uniform-expert MoE under the same computational budget. They also apply a P-Penalty Loss, which penalizes the model's tendency to favor large experts and encourages it to activate more small ones. Whether this generalizes is an open question, but it's a concrete departure from the uniform-expert dogma that DeepSeek and Mixtral popularized.