TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

Multi-Device AI Inference with TensorRT

5 posts · May 2026 – Oct 2026

What this trail covers

This feature thread tracks the evolution of enabling high-performance AI inference across multiple GPUs and edge devices. Initial efforts focused on the challenges of scaling generative AI workloads beyond single-GPU memory and compute limits. TensorRT 11.0 introduced native multi-device inference support, leveraging NVIDIA NCCL for distributed collectives and enabling seamless scaling of generative AI pipelines. This includes the implementation of context parallelism strategies like AllGather K. The DIN Deploy samples extend this by providing C++ implementations for local AI applications using ONNX Runtime and TensorRT RTX, demonstrating performance gains for various models and enabling graphics interop for advanced use cases.

All NVIDIA trails
  1. Build Local AI Apps with C++ and NVIDIA TensorRT RTX Samples | NVIDIA Technical BlogThis post introduces the Do Inference Now (DIN) Deploy open-source C++ samples, which combine ONNX Runtime with the NVIDIA TensorRT RTX execution provider for accelerated local AI inference on Windows and Linux. It details a workflow separating model export (via Python exporter to ONNX) from deployment (native C++ CLI using ONNX Runtime session and tensor APIs). Performance measurements on DGX Spark show significant GPU acceleration for ASR and image generation tasks. The FLUX.2 sample highlights graphics interop with Vulkan/DirectX and the use of post-training quantization with NVIDIA Model Optimizer for drop-in ONNX replacements. The samples provide CMake presets for various platforms and demonstrate integration with models like OpenAI Whisper, NVIDIA Parakeet TDT, NVIDIA Nemotron ASR Streaming, Meta SAM 2.1, and FLUX.2-klein-4B.
  2. Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton | NVIDIA Technical BlogThis post details the integration of NVIDIA TensorRT multi-device inference with NVIDIA Dynamo-Triton (formerly Triton Inference Server) release 26.07. It demonstrates how a single Triton model instance can leverage multiple GPUs for distributed inference, specifically using Ulysses context parallelism for the NVIDIA Cosmos 3 Nano video generation model. The post quantifies the end-to-end latency reduction from 156.6 seconds on a single GPU to 34.2 seconds on eight GPUs, with a transformer RPC speedup of 6.09x. It explains how the distributed graph is compiled into TensorRT plans and loaded by Dynamo-Triton, exposing a single gRPC endpoint to the client. Configuration details for enabling multi-device inference in Triton are provided, along with a breakdown of the Ulysses context parallelism strategy and benchmarking results.
  3. Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine | NVIDIA Technical BlogThis post details the integration of NVIDIA Transformer Engine with JAX to accelerate dropless MoE training. Key contributions include: 1) Achieving a 10.4x throughput improvement for MoE training on NVIDIA GB200 by optimizing DeepSeek-V3 performance from 103 to 1,068 TFLOPS/GPU. 2) Implementing dropless MoE using grouped GEMM kernels that handle variable expert token counts in a single call, preserving model quality without token dropping or padding. 3) Accelerating expert parallelism operations with NCCL EP, which fuses dispatch and combine stages and deduplicates tokens to reduce network traffic. 4) Incorporating additional optimizations like JAX host offloading and XLA multistreaming collectives to reduce memory bottlenecks and overlap communication. 5) Demonstrating sustained 97% scaling efficiency at 1,024 GPUs on NVIDIA GB300 NVL72 hardware for training DeepSeek-V3 671B.
  4. When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving | NVIDIA Technical BlogThis post introduces and details the encode-prefill-decode (EPD) disaggregation technique within NVIDIA Dynamo for accelerating multimodal model serving. It explains the rationale behind EPD by analyzing the latency breakdown in aggregated serving for multimodal requests, highlighting how vision encoding can become a bottleneck. The post describes three encoder placement topologies: aggregated, colocated encoder, and disaggregated encoder, and discusses the trade-offs and scenarios where each is most beneficial. It quantifies the performance gains (up to 5x faster TTFT, 7x faster E2E response) and identifies key factors influencing these gains, such as input media load, output sequence length, model size/precision, and traffic mix. Specific test environment details and experimental results are provided, demonstrating the impact of quantization and placement strategies on goodput.
  5. Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson | NVIDIA Technical BlogThis post details the deployment and optimization of compact open models (Nemotron 3.5 Lightning, Qwen3.8-27B) on NVIDIA Jetson platforms. It introduces NVFP4 quantization and speculative decoding (DSpark, DFlash2) as key techniques to improve inference performance, demonstrating up to 6.28x decode throughput speedup over BF16. It highlights the importance of choosing models based on workload characteristics (MoE vs. dense) and testing specific optimization configurations for different models. The post also emphasizes application-level validation with representative prompts to ensure critical behaviors are preserved.
  7. Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference | NVIDIA Technical BlogIntroduces speculative decoding as a technique to accelerate LLM inference by using a draft model to propose tokens for parallel verification by a larger target model. Provides five guidelines for selecting optimal draft length and draft mechanisms, focusing on balancing compute-bound regions, attention performance, and latency overhead. Compares various draft mechanisms and introduces SPEED-Bench for evaluating speculative decoding performance.
  8. ModelExpress: Distributing Model Artifacts at the Speed of Light | NVIDIA Technical BlogThis post introduces NVIDIA ModelExpress (MX), a system designed to accelerate the distribution of model artifacts, particularly model weights and kernel caches, for AI inference. MX prioritizes direct GPU-to-GPU P2P RDMA transfers via NIXL, bypassing object storage and host memory when compatible peers are available. For initial worker bootstrapping, MX supports streaming from object storage (via ModelStreamer, avoiding local disk) and direct loading from local storage (via GDS or ModelStreamer, bypassing host memory staging). MX also implements an atomic distributed caching mechanism for cluster ingress to ensure model weights are downloaded only once. The system uses a control plane for peer discovery (via Redis or Kubernetes metadata) and a data plane leveraging NIXL for efficient transfers across various network types. Optimizations for NIXL memory registration overhead are also discussed.
  9. Accelerating End-to-End Co-Folding Performance with NVIDIA BioNeMo Agent Toolkit | NVIDIA Technical BlogThis post details the acceleration of end-to-end biomolecular structure prediction pipelines using the NVIDIA BioNeMo Agent Toolkit. It highlights the performance gains of MMseqs2-GPU for MSA generation (up to 177x faster than CPU JackHMMER), the latency reduction and extended sequence length support provided by cuEquivariance for OpenFold3 inference (up to 3x faster, ~5.9k tokens), and the further optimizations in the OpenFold3 NIM (up to 6,400 tokens on a single B300). It also introduces Fold-CP for context-parallel inference, enabling the modeling of large assemblies (up to 32,000 tokens on 64 B300 GPUs).
  10. Scaling AI Inference Across Multiple GPUs Using NVIDIA TensorRT with Multi-Device Inference Support | NVIDIA Technical BlogThis post details the introduction of native multi-device inference support in TensorRT 11.0, enabling scaling of generative AI pipelines across multiple GPUs. It explains how TensorRT leverages NVIDIA NCCL for distributed collectives and introduces context parallelism strategies like AllGather KV, Ring Attention, and DeepSpeed Ulysses. The post provides technical details on how these strategies partition input sequences across GPUs to address the memory and compute challenges of long-sequence attention workloads, and presents benchmark results comparing their performance.
  11. Q&A: How KRAFTON Built PUBG Ally, a Co-Playable Character Powered by NVIDIA ACE | NVIDIA Technical BlogThis post details KRAFTON's implementation of PUBG Ally, a co-playable character in PUBG: Battlegrounds, which utilizes NVIDIA ACE for on-device AI inference. Key technical contributions include: 1. Architectural separation of fast, reactive gameplay (behavior tree) from slower, language-driven reasoning (SLM) to achieve real-time combat responsiveness. 2. Use of a 2B parameter Mistral-NeMo-Minitron-2B SLM, further quantized, to fit within limited GPU VRAM (8GB minimum) and minimize latency by running on-device, removing network round-trip delays. 3. Prompt engineering techniques to optimize KV cache usage by maintaining stable instructions and context while updating only relevant real-time information. 4. Domain adaptation for PUBG-specific terminology, maps, items, and weapon attributes through constrained world definition (single map, single mode, fixed item taxonomy) and grounding responses in live game state observation. 5. Integration of ASR, SLM, and TTS components within an agent harness for a continuous, interactive pipeline.
  12. Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding | NVIDIA Technical BlogIntroduces DFlash, an open-source block diffusion model for speculative decoding, which significantly accelerates LLM inference on NVIDIA Blackwell GPUs by drafting entire token blocks in parallel. Benchmarks demonstrate up to 15x throughput improvement for gpt-oss-120b and nearly double interactivity for Llama 3.1 8B compared to EAGLE-3. DFlash is integrated into vLLM and SGLang, enabling adoption without application refactoring, and has released 20 model checkpoints on Hugging Face.
  13. Run DiffusionGemma on NVIDIA for Developer-Ready, High-Throughput Text Generation | NVIDIA Technical BlogThis post details the optimization and deployment of DiffusionGemma, a novel diffusion-based text generation model, on various NVIDIA platforms. It highlights the model's efficiency on data center GPUs (H100, DGX Spark, DGX Station) and client GPUs (RTX/RTX PRO), achieving high-throughput token generation through parallel processing. It introduces deployment methods via Hugging Face, NVIDIA NIM, and NeMo AutoModel for prototyping and enterprise use, including specific guidance on using vLLM for multi-user serving and NIM for containerized inference microservices.
  14. NVIDIA Blackwell Sets STAC-AI Record for LLM Inference in Finance | NVIDIA Technical BlogThis post details the performance results of NVIDIA Blackwell platforms (HGX B200, RTX PRO 6000 Blackwell) on the STAC-AI LANG6 benchmark for LLM inference in financial use cases. It showcases significant speedups in batch and interactive modes, with up to 2.8x improvement on HGX B200 compared to GH200. The post highlights the use of TensorRT LLM with FP8 and NVFP4 quantization, and discusses the impact of chat templates and tokenization during inference. It also provides links to detailed STAC reports for HPE, Lambda, and Supermicro systems.