TechPrufer
Get started
Glean
Glean logoGlean · Feature trail

AI Chat Latency Optimization

4 posts · Nov 2025 – Aug 2026

What this trail covers

Glean's AI chat capabilities are being optimized for latency and cost efficiency. This post investigates the impact of input token count and KV caching on Time To First Token (TTFT) and overall chat latency for LLM calls. KV caching significantly reduces latency by reusing computations across LLM calls, especially in multi-turn conversations or workflows with similar prefixes. Experimentation with Azure's KV cache shows a ~0.15ms saving per cached input token, leading to substantial TTFT reduction. The post also discusses RAG vs. fine-tuning, their benefits, use cases, and challenges, including data privacy, potential limitations, and infrastructure complexity. A hybrid approach combining RAG and fine-tuning is also explored.

All Glean trails
  1. Right-sizing intelligence: Glean saves 81% on token costs and is preferred 78% of the time versus Claude CoworkThis post details Glean's approach to optimizing AI chat latency and cost by implementing a Pareto frontier analysis to map cost and performance tradeoffs across various models. It introduces Glean's auto-routing mechanism, which matches tasks to models and their reasoning levels based on effort and capability required, using a specialist model (Glean Waldo) for reasoning level determination. The post also presents benchmark analysis showing Glean's 81% cost savings and 78% preference rate over Claude Cowork, attributing this to architectural decisions like a unified enterprise context and an efficient harness design that minimizes token usage. It highlights how Glean's preferred responses reduce 'botsitting' by providing more accurate, complete, and grounded outputs.
  2. A complete guide to retrieval augmented generation vs fine-tuningThis post provides a comprehensive comparison of Retrieval Augmented Generation (RAG) and fine-tuning for enhancing Large Language Models (LLMs). It details the mechanics of RAG (retrieval, augmentation, generation) and its types (traditional, agentic), along with its benefits (real-time insights, reduced hallucinations, flexibility). It also explains fine-tuning, its types (full, PEFT, continuous pretraining), and benefits (domain expertise, maintained general capabilities). Key differences in purpose, implementation, and cost are highlighted, along with specific use cases for each. The post further explores the benefits and considerations of a hybrid RAG and fine-tuning approach, and discusses challenges associated with RAG such as data privacy, security, potential limitations, biases, and validation complexities. While the post mentions latency optimization in the context of AI chat, the primary technical contribution is the comparative analysis of RAG and fine-tuning methodologies.
  3. How input token count impacts the latency of AI chat toolsThis post details an empirical study on the correlation between input token count and Time To First Token (TTFT) for LLM calls, specifically using GPT-4 Turbo. It explains the quadratic time complexity of the Transformer's attention mechanism with respect to context window size and presents data showing a linear relationship between input tokens and TTFT, with an approximate increase of 0.24ms per input token for P95 TTFT. The analysis also contrasts this with end-to-end latency improvements and demonstrates how splitting large prompts can significantly reduce perceived latency.
  4. How KV caches impact time to first token for LLMsThis post details the impact of KV (key-value) caching on LLM Time To First Token (TTFT) latency and throughput. It explains how KV caching, by storing intermediate attention computations, reduces the complexity of subsequent token generation from quadratic to linear. The post presents experimental results showing that reusing KV caches across LLM calls can save approximately 0.15ms per cached input token, leading to significant TTFT improvements (e.g., 100ms for 1000 cached tokens). This optimization is crucial for delivering a faster and more responsive Glean Assistant experience.
  5. LLM choice in Assistant: A better agentic coworker sourcing deep proficiencies from your models of choiceIntroduces LLM choice for Glean Assistant, allowing users to select AI models optimized for specific tasks (e.g., coding, creative assets, general purpose) to improve accuracy and efficiency. This feature provides flexibility by integrating with various LLM providers without requiring users to switch tools or sign new contracts. It also enhances enterprise oversight by allowing administrators to curate available model families, define usage policies, and govern access. The post highlights the benefits of matching AI models to work requirements for better outcomes and avoiding vendor lock-in.
  7. 5 metrics to measure AI-generated answers' decision-making impactThis post introduces five key metrics for measuring the decision-making impact of AI-generated answers: accuracy, relevance, coherence, helpfulness, and user trust. It details technical approaches for measuring each, including factual error rate, faithfulness scores, BLEU/ROUGE for accuracy; user rating scales and structured scoring rubrics for relevance; perplexity and readability scores, along with human review checklists, for coherence; and user workflow understanding for helpfulness. It also discusses the importance of grounding AI answers in verified company data via RAG for accuracy and compliance.