Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine | Modal Blog

Introduced Quail, a query-aware inference layer that integrates a query planner with an inference engine. Achieved over 1 billion tokens per minute per H100 GPU, outperforming vLLM by 10x on specific AI-SQL benchmarks. The core innovation lies in leveraging query plans to optimize KV cache usage and reduce host overhead, enabling more efficient processing of large-scale, structured inference workloads.