ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale

What this post added

Introduces ThunderAgent, a system that enhances agentic inference by treating agent workflows as schedulable programs rather than individual requests. This program-level abstraction addresses KV cache thrashing by selectively pausing low-priority workflows under memory pressure and routing resumed workflows to nodes with available capacity. ThunderAgent improves single-node throughput by up to 2.5x and achieves near-linear scaling on multi-node clusters, offering a drop-in solution compatible with existing inference engines and optimizations.