Skip to main content
Back

NVIDIA's Apex Finally Fixes Broken pip Builds and Adds Blackwell GPU Support

NVIDIA's Apex PyTorch extension adds environment-variable-based builds, parallel compilation, and Blackwell GPU support in its latest push

Repo, Full Access
  • NVIDIA/apex adds environment-variable build flags (APEX_CPP_EXT, APEX_CUDA_EXT) to fix pip 23.1+ compatibility issues.
  • New APEX_PARALLEL_BUILD variable enables multi-job compilation, significantly cutting build times for full extension builds.
  • Blackwell GPU (compute capability 10.0, 12.0) cross-compilation targets added automatically when CUDA >= 12.8.
  • Legacy --global-option flags remain supported for backward compatibility with existing build scripts.
  • Fused kernels for Adam, LayerNorm, RMSNorm, and DistributedDataParallel remain Apex's core value for large-scale LLM training.
  • Available now on GitHub; pre-built in the NVIDIA NGC PyTorch container (26.05-py3).

NVIDIA/apex, the long-running PyTorch extension for mixed precision and distributed training, just received a meaningful set of updates to its build system. The changes land in the master branch and address a friction point that has frustrated Apex users for years: the old --global-option install flags broke with newer versions of pip, and building the library's many CUDA extensions was painfully slow with no parallelism. Both problems now have official fixes.

A build system that finally works with modern pip

The core change is a new environment-variable-based build interface. For performance and full functionality, NVIDIA now recommends installing Apex with CUDA and C++ extensions using environment variables. Instead of the old --global-option flags that pip deprecated and eventually broke, you now set shell variables before calling pip:

git clone https://github.com/NVIDIA/apex
cd apex
# Recommended: build with core C++ and CUDA extensions
APEX_CPP_EXT=1 APEX_CUDA_EXT=1 pip install -v --no-build-isolation .
# Build everything, including contrib extensions
APEX_CPP_EXT=1 APEX_CUDA_EXT=1 APEX_ALL_CONTRIB_EXT=1 pip install -v --no-build-isolation .