The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

This post introduces FriendliAI's approach to building an inference cloud for AI agents. Key technical contributions include: 1. Prefix Caching: Computing shared prefixes of agent steps only once to avoid redundant computation. 2. Hierarchical KV Cache: Utilizing GPU, host memory, and disk for KV caching to manage large contexts efficiently. 3. Cache-Aware Routing: Directing agent requests to replicas that already hold the necessary prefix, preserving locality. 4. Agent-Aware Scheduling: Recognizing that individual calls are part of a larger agent program to optimize execution. These techniques are presented as a new paradigm for agentic inference, contrasting with traditional chat-based inference economics.