Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave

This post details CoreWeave's approach to serving diverse inference workloads, from MVP to trillion-parameter models. Key technical contributions include: prioritizing KV cache locality and least loaded GPUs in routing across heterogeneous hardware; offloading KV cache to high-bandwidth storage between chat turns to avoid re-paying for prefill; and leveraging 4-bit quantization and custom-trained speculators for improved acceptance lengths. The platform supports both serverless and dedicated consumption models, adapting to workload shapes like agentic, chat, voice/video streaming, and batch processing.