Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton | NVIDIA Technical Blog

This post details the integration of NVIDIA TensorRT multi-device inference with NVIDIA Dynamo-Triton (formerly Triton Inference Server) release 26.07. It demonstrates how a single Triton model instance can leverage multiple GPUs for distributed inference, specifically using Ulysses context parallelism for the NVIDIA Cosmos 3 Nano video generation model. The post quantifies the end-to-end latency reduction from 156.6 seconds on a single GPU to 34.2 seconds on eight GPUs, with a transformer RPC speedup of 6.09x. It explains how the distributed graph is compiled into TensorRT plans and loaded by Dynamo-Triton, exposing a single gRPC endpoint to the client. Configuration details for enabling multi-device inference in Triton are provided, along with a breakdown of the Ulysses context parallelism strategy and benchmarking results.