Deploying an HSTU Generative Recommender with NVIDIA Dynamo-Triton | NVIDIA Technical Blog

This post details the deployment of an HSTU Generative Recommender using NVIDIA Dynamo-Triton. It introduces the integration of PyTorch Ahead-of-Time (AOTI) Inductor compilation to export HSTU models into native C++ artifacts, reducing Python runtime overhead. It also highlights the use of FlexKV-backed KV caching to store reusable attention state and avoid recomputing long user histories, demonstrating up to 5.93x lower latency on an NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU at dynamic batch size 8 with a 100% GPU KV-cache hit rate. The workflow combines custom operator builds, AOTI export, FlexKV service startup, native C++ validation, and Dynamo-Triton deployment.