Skip to main content
Back

NVIDIA Shrinks Mistral-Medium-3.5 to Half Its Size Without Losing Accuracy

NVIDIA quantizes Mistral's 128B unified model to FP4, slashing memory by 1.8x vs FP8 with near-zero accuracy loss on Blackwell GPUs

Model, Full Access
  • NVIDIA released Mistral-Medium-3.5-128B-NVFP4, a 4-bit quantized version of Mistral's 128B flagship model, ready for Blackwell GPU inference.
  • Near-zero accuracy loss: NVFP4 scores within 1% of FP8 across all benchmarks, and actually outperforms FP8 on long-context recall (AA-LCR: 65.10% vs 62.06%).
  • 1.8x memory reduction vs FP8, 3.5x vs FP16, enabling the 128B model to run on fewer Blackwell GPUs with higher throughput.
  • Surgical quantization: Only MLP layers 4-86 go to FP4; edge layers, attention, and KV cache stay in FP8 to protect accuracy.
  • Blackwell-only: NVFP4 requires native FP4 Tensor Cores (RTX 50-series / B200); H100 and A100 get no speedup from this format.
  • The base model scores 77.6% on SWE-Bench Verified and 88.75% on AIME 2025, replacing Devstral 2 and Magistral as Mistral's unified flagship.

NVIDIA has released Mistral-Medium-3.5-128B-NVFP4 on Hugging Face: a post-training quantized version of Mistral's flagship 128B model, compressed down to 4-bit floating point precision using NVIDIA's NVFP4 format. The checkpoint is ready for inference on Blackwell-class GPUs via vLLM, and it's available now for commercial and non-commercial use.

One model to rule them all

Mistral Medium 3.5 is a dense 128B model with a 256k context window, handling instruction-following, reasoning, and coding in a single set of weights. That last part is the real story. The dedicated reasoning model (Magistral) and the dedicated coding model (Devstral 2) are both retired into a single dense 128B set of weights. One model, three jobs, configurable reasoning effort per request.

The 77.6% SWE-bench Verified score beats Devstral 2 (72.2%) by over 5 points and surpasses Qwen 3.5 397B despite being a much smaller dense model. It also scores 91.4% on the Tau3-Telecom agentic benchmark. Medium 3.5 supports configurable reasoning effort per request , you control how much "thinking" the model does before responding.

What NVFP4 actually is

NVFP4 is a 4-bit floating-point format purpose-built for NVIDIA's Blackwell GPU generation. The core challenge with any 4-bit format is that you only have 16 representable values, so naively rounding weights to 4 bits destroys accuracy. NVFP4 solves this with a two-level scaling strategy.

  • Per-block FP8 scale: Every 16 weights share one FP8 (E4M3) scaling factor, which adapts to the local distribution of those values.
  • Per-tensor FP32 scale: A single high-precision global scalar normalizes the entire tensor, giving the per-block scales room to work accurately.