Inference Latency: What It Measures & Why It Varies

This post defines and differentiates various inference latency metrics (TTFT, TPOT, total response time, agent run time) and explains why averages and model-only benchmarks can be misleading. It details factors affecting latency beyond the model itself, including queuing, guardrails, cold starts, and the impact of retrieval in RAG pipelines. The post then discusses which latency metric is most relevant for different workloads (chat/streaming vs. agents/batch jobs) and introduces semantic caching as a strategy to reduce latency by avoiding model calls. It highlights Redis Iris and Redis LangCache as solutions for low-latency retrieval, caching, and state management in AI applications, emphasizing their role in optimizing latency budgets for chat, agents, and batch jobs.