Enabling Private High-Performance Production AI Inference with NVIDIA Confidential Computing | NVIDIA Technical Blog

This post details the performance impact of NVIDIA Confidential Computing (CC) on TensorRT LLM inference using NVIDIA Blackwell GPUs. It quantifies CC overhead by comparing CC-on and CC-off runs, showing retained throughput of 96.1-98.2% and TPOT overhead of 1.2-4.3%. It identifies specific CC-aware adaptations made in TensorRT LLM to mitigate performance degradation, including CC-aware memory selection for host-to-device transfers, asynchronous device-to-host readback, using the GPU %globaltimer for kernel autotuner timing, and choosing CC-aware multi-GPU communication algorithms. The post provides a methodology for measuring CC overhead on custom workloads.