TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

LLM Serving Simulation and Optimization

2 posts · May 2026 – Sep 2026

What this trail covers

This feature thread tracks the evolution of tools and techniques for simulating and optimizing Large Language Model (LLM) serving stacks. Initial efforts focused on the complexity of tuning LLM deployments due to interacting choices in model backends, tensor parallelism, scheduling, routing, and caching. DynoSim has been introduced as a workload-driven discrete-event simulation of the NVIDIA Dynamo serving stack, combining measured engine timings, scheduler behavior, router and planner logic, KV cache performance, and GPU telemetry. AIPerf is a new multiprocess benchmarking tool that replaces GenAI-Perf, offering more accurate server benchmarking by preventing client-side bottlenecks. It supports a wide range of endpoint types, public datasets, and configurable arrival patterns for realistic workload testing. Core metrics like TTFT, ITL, request latency, and output token throughput are reported with percentile breakdowns and GPU telemetry.

All NVIDIA trails
  1. Benchmarking LLM Inference at Scale with AIPerf | NVIDIA Technical BlogThis post introduces AIPerf, a new multiprocess benchmarking tool that replaces GenAI-Perf. AIPerf addresses client-side bottlenecks by using a multiprocess architecture with separate worker processes for load generation and record-processor services for results handling, coordinated via ZMQ. It supports over 15 endpoint types and various dataset formats for realistic workload testing, along with configurable arrival patterns (constant, Poisson, gamma) and burstiness. The tool reports core metrics like TTFT, ITL, request latency, and output token throughput with percentile breakdowns and GPU telemetry (when DCGM or pynvml is available). The post provides a walkthrough of running a synthetic ISL/OSL benchmark on vLLM with Qwen3-0.6B, detailing installation and command-line usage.
  2. How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra | NVIDIA Technical BlogThis post details how full-stack NIM optimizations, specifically for Nemotron 3 Ultra on a 4xB200 system, achieve up to 2.5x higher system throughput compared to a baseline serving stack. The optimizations include autotuned kernels for MoE and Mamba architectures, tensor parallelism, prefix and model-state reuse (including partial-prefix matching), scheduler and batching tuning, and MTP speculative decoding. It also introduces AIPerf for benchmarking representative workloads and selecting Pareto points that meet latency SLOs, and provides guidance on deploying the Nemotron 3 Ultra NIM with specific profiles and speculative decoding enabled.
  3. Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference | NVIDIA Technical BlogThis post introduces speculative decoding as a technique to accelerate LLM inference by having a smaller draft model propose tokens that a larger target model verifies in parallel. It provides five guidelines for selecting the optimal draft length and mechanism, focusing on pushing GEMMs into the compute-bound region and optimizing attention performance based on query head count. The post also introduces SPEED-Bench, a benchmark for measuring speculative decoding performance, and highlights available training examples in NVIDIA/Model-Optimizer for draft mechanisms like EAGLE-3, DFlash, and DSpark.
  4. Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo | NVIDIA Technical BlogThis post introduces and details the implementation of Shadow Engine Recovery in NVIDIA Dynamo. It addresses the slow recovery times of LLM inference engines after process failures by keeping a fully initialized shadow engine idle on the same GPUs as the active engine. Key technical contributions include the GPU Memory Service (GMS) for persistent GPU memory and weight sharing, and the concept of pre-initialized shadow engines that defer non-transferable initialization until activation. The post quantifies the performance improvement, showing a reduction in recovery time from 283 seconds to 7.3 seconds for a GLM-5.2 deployment. It also outlines the integration with inference frameworks like vLLM via `torch.cuda.CUDAPluggableAllocator` and discusses future development for KV cache sharing.
  5. DynoSim: Simulating the Pareto Frontier | NVIDIA Technical BlogThis post introduces DynoSim, a workload-driven discrete-event simulation of the NVIDIA Dynamo serving stack. DynoSim models the interactions between model backends, schedulers, routers, planners, and KV cache behavior to enable fast simulation of LLM serving configurations. It uses measured engine timings (via AIC) and component behaviors to simulate request journeys on a virtual timeline, achieving up to 1500x speedup over real-time. The post details DynoSim's architecture, its discrete-event simulation approach, and demonstrates its utility in mapping the Pareto frontier for LLM serving by comparing round-robin routing with KV-aware routing, showing improvements in prefix cache reuse, TTFT, and throughput.