Scaling AI Inference Across Multiple GPUs Using NVIDIA TensorRT with Multi-Device Inference Support | NVIDIA Technical Blog

What this post added

This post introduces native multi-device inference support in NVIDIA TensorRT 11.0, enabling scalable AI inference across multiple GPUs and edge devices. It details the integration with NVIDIA NCCL for high-throughput distributed collectives and explores context parallelism strategies (AllGather KV, Ring Attention, DeepSpeed Ulysses) for optimizing long-sequence attention workloads. Benchmarks highlight DeepSpeed Ulysses' performance for diffusion models at extreme context lengths.