Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo | NVIDIA Technical Blog

Introduces Shadow Engine Recovery in NVIDIA Dynamo, a feature that keeps a fully initialized shadow engine idle on the same GPUs as the active engine. Leverages the GPU Memory Service (GMS) to share weights without duplicating them in HBM, enabling recovery within seconds. Demonstrates a 39x speedup in recovery time for a GLM-5.2 deployment compared to a cold restart. Details the GMS architecture using CUDA Virtual Memory Management API and its integration with inference frameworks like vLLM. Explains the concept of parked shadow engines that defer KV cache materialization.