Scaling AI Inference Across Multiple GPUs Using NVIDIA TensorRT with Multi-Device Inference Support | NVIDIA Technical Blog

This post details the introduction of native multi-device inference support in TensorRT 11.0, enabling scaling of generative AI pipelines across multiple GPUs. It explains how TensorRT leverages NVIDIA NCCL for distributed collectives and introduces context parallelism strategies like AllGather KV, Ring Attention, and DeepSpeed Ulysses. The post provides technical details on how these strategies partition input sequences across GPUs to address the memory and compute challenges of long-sequence attention workloads, and presents benchmark results comparing their performance.