Why Inference Latency, Availability Drift | CoreWeave Blog

This post details the challenges of inference latency and availability drift in production, focusing on the gradual degradation that often goes unnoticed by standard monitoring. It identifies three primary causes for latency drift: infrastructure-layer variability (GPU contention, scheduling overhead), model-serving configuration drift (batching, KV cache pressure), and traffic pattern mismatch (autoscaling lag, request shape). The post introduces key metrics for diagnosing latency drift, including p99 latency, time-to-first-token (TTFT), and goodput vs. throughput. It further explains how availability degrades without outright failure through elevated error rates, autoscaling lag, and stale cache behavior, proposing metrics like error budget consumption rate, autoscaling lag vs. burst profile, and timeout rate under load to monitor availability degradation. The core contribution is a framework for understanding and measuring the subtle, drift-based failures common in large-scale inference systems.