Baidu's PaddlePaddle 3.3 Beats FlexAttention by 2.1x With FlashMaskV3
Baidu's PaddlePaddle 3.3 ships FlashMaskV3, a new checkpoint system, and dynamic GPU memory defragmentation for large-model training
Takeaways
- FlashMaskV3 ships with a new load-balancing GPU scheduler, claiming up to 2.1x speedup over PyTorch's FlexAttention and 80% over Megatron-LM in distributed settings.
- FlexCheckpoint + AOA introduces a mini-language for describing weight resharding, automating conversion between training and inference parallelism configs, with HuggingFace format support.
- VMM Allocator performs online GPU memory defragmentation, reducing fragmentation in MoE model training from 10%+ down to 3%.
- A new paddle.compat API namespace and operator compatibility layer make it easier to integrate third-party kernels like FlashInfer, DeepGEMM, and SonicMoE.
- New debugging tools include forward/backward graph visualization, scoped logging, and per-tensor MD5 checksums for numerical reproducibility checks.
- Available now on GitHub under Apache 2.0; install via the official quick install page.
PaddlePaddle, Baidu's open-source deep learning framework, has published its 3.3.0 release on GitHub. The update is squarely aimed at the pain points of large-model training at scale: attention kernel performance, the cost of converting weights between training and inference configurations, and GPU memory fragmentation in Mixture-of-Experts (MoE) models. It also ships a new compatibility layer designed to make it easier to drop in third-party high-performance kernels without rewriting model code.
FlashMaskV3: Beating FlexAttention on Sparse Attention
The headline kernel upgrade is FlashMaskV3, PaddlePaddle's column-sparse attention mask implementation. Sparse attention masks let you express patterns like causal masking, sliding-window attention, or document-boundary masking without materializing a full dense attention matrix , which matters a lot for long-context training.
The latest version, v3.3.1, was last published in March 2026. The 3.3.0 release notes describe the core innovation in FlashMaskV3 as a new Persistent Preemptive Tile (PPT) scheduler , a GPU kernel scheduling strategy that balances compute load across streaming multiprocessors (SMs) at runtime rather than statically. The claimed results are significant:
- Up to 2.1x faster than PyTorch's FlexAttention on a single GPU
- Up to 1.4x faster than the previous FlashMask version
- 80% faster than Megatron-LM's distributed attention implementation in context-parallel scenarios
The kernel also natively supports context parallelism (splitting long sequences across GPUs) and head dimensions larger than 128, which is relevant for models using grouped-query attention with wide heads.