Skip to main content
Back

NVIDIA's Transformer Engine Ships 4-Bit Training That Beats FP8 by 1.73x

Transformer Engine v2.15 ships FlashAttention 4, MXFP8 attention, and deeper NVFP4 support for Blackwell GPUs — pushing 4-bit pretraining closer to production

Repo, Full Access
  • v2.15 ships: NVIDIA's Transformer Engine v2.15 adds FlashAttention 4, MXFP8 attention, and expanded NVFP4 support.
  • NVFP4 optimizer: NVFP4 weight quantization is now supported in the fused Adam optimizer, enabling 4-bit weights during training steps.
  • MoE coverage: New NVFP4 and MXFP8 quantization kernels for grouped tensors cover Mixture-of-Experts architectures on Blackwell GPUs.
  • 7x GEMM speedup: NVFP4 delivers 7x GEMM throughput over Hopper FP8 on GB300, with up to 1.73x speedup over FP8 baselines end-to-end.
  • Accuracy validated: A 12B model trained over 10 trillion tokens with NVFP4 matched FP8 loss curves and downstream benchmarks throughout.
  • Free and open: Available via pip install transformer_engine[pytorch] under Apache 2.0; Blackwell features require CUDA 12.8+.

NVIDIA's Transformer Engine has shipped v2.15, the latest release of its open-source library for accelerating Transformer models using reduced-precision arithmetic. The headline additions are FlashAttention 4 support, MXFP8 attention, and a wave of improvements to NVFP4 , NVIDIA's 4-bit floating-point format , that collectively push the library's Blackwell GPU story forward in a meaningful way.

Why precision formats matter this much

The core idea behind Transformer Engine is simple: instead of running every matrix multiplication in 32-bit or 16-bit floating point, you use narrower formats that pack more operations into the same hardware. Mixed-precision training delivers significant speedups with minimal impact on accuracy, and FP8 , introduced with the Hopper GPU architecture , offers further performance gains over FP16 with no degradation in accuracy. Blackwell takes this further. On Blackwell GPUs, TE also supports MXFP8 (Microscaling FP8) and NVFP4 formats for even greater efficiency.

NVFP4 is the most aggressive format in the stack: 4 bits per value, compared to 8 for FP8 and 16 for BF16. Native hardware support of NVFP4 on the NVIDIA GB300 Grace Blackwell Ultra Superchip delivers 7x GEMM throughput compared to native FP8 precision on the NVIDIA Hopper. That kind of raw compute uplift is why the library has been steadily expanding its NVFP4 coverage across releases.

What's new in v2.15

The v2.15 release was tagged in May 2026. The key additions span attention, MoE (Mixture-of-Experts), and optimizer support:

  • FlashAttention 4: PyTorch support for Flash Attention 4 was added. FA4 is the latest iteration of the fused attention kernel that avoids materializing the full attention matrix in memory.
  • MXFP8 attention: Attention can now run in MXFP8 precision, extending low-precision coverage beyond just the linear layers.