Skip to main content
Back

NVIDIA Upgrades nccl-tests to Benchmark GPU-Initiated Networking on Multi-GPU Clusters

NVIDIA updates nccl-tests with flags to benchmark the new device-side communication APIs and symmetric memory introduced in NCCL 2.28+

Repo, Full Access
  • NVIDIA updated nccl-tests to add flags for benchmarking NCCL's new device-side communication APIs and symmetric memory.
  • New -D flag selects device API implementations for all_reduce and alltoall; requires symmetric buffer registration via -R 2.
  • New -x flag enables zero-CTA copy-engine collectives that free streaming multiprocessors for compute overlap.
  • NCCL 2.30.7 released the same day, targeting CUDA 13.3, with device API and symmetric kernel improvements.
  • Device API supports three modes: LSA (NVLink P2P), Multimem (NVLink SHARP multicast), and GIN (GPU-initiated networking over InfiniBand/RoCE).
  • New -J (JSON output), -T (timeout), and -M (memory report) flags improve CI integration and large-scale debugging.

NVIDIA has pushed a significant update to nccl-tests, its open-source benchmarking suite for the Collective Communications Library (NCCL). The update lands alongside NCCL's rapid evolution through versions 2.28 to 2.30, and it adds a set of new CLI flags that let you directly benchmark the library's newest and most impactful features: device-initiated communication and symmetric memory.

Why this update matters now

NCCL has been undergoing a quiet architectural shift. NCCL 2.28 focused on GPU-initiated networking, device APIs for communication-compute fusion, copy-engine-based collectives, and new APIs for developers to build efficient, scalable distributed applications. That's a meaningful change from the traditional model where all NCCL operations were host-initiated, which introduced synchronization overhead between the CPU and GPU.

With the new device API, kernels initiate data movement directly, integrating communication with compute operations for more throughput and less overhead. To use the new API, you must use data buffers with symmetric memory windows. Symmetric memory here means buffers that are registered across all ranks so every GPU can resolve the remote address of any other GPU's buffer without going through the host.

The problem was that nccl-tests had no way to exercise these new paths. The update fixes that.

The new flags, explained

The updated nccl-tests exposes several new arguments that map directly onto NCCL's new capabilities: