Build Local AI Apps with C++ and NVIDIA TensorRT RTX Samples | NVIDIA Technical Blog

This post introduces the Do Inference Now (DIN) Deploy open-source C++ samples, which combine ONNX Runtime with the NVIDIA TensorRT RTX execution provider for accelerated local AI inference on Windows and Linux. It details a workflow separating model export (via Python exporter to ONNX) from deployment (native C++ CLI using ONNX Runtime session and tensor APIs). Performance measurements on DGX Spark show significant GPU acceleration for ASR and image generation tasks. The FLUX.2 sample highlights graphics interop with Vulkan/DirectX and the use of post-training quantization with NVIDIA Model Optimizer for drop-in ONNX replacements. The samples provide CMake presets for various platforms and demonstrate integration with models like OpenAI Whisper, NVIDIA Parakeet TDT, NVIDIA Nemotron ASR Streaming, Meta SAM 2.1, and FLUX.2-klein-4B.