Beyond prompt engineering: the real drivers of token efficiency in enterprise AI

This post argues that token efficiency in enterprise AI is primarily driven by retrieval quality, context selection, and orchestration design, rather than solely prompt engineering. It details how Glean's Work AI platform leverages Enterprise Context to improve retrieval by connecting knowledge across systems and respecting permissions. The post outlines a four-layer framework for token efficiency: retrieving better and less, passing only relevant evidence through chunking and passage selection, structuring prompts for control, and orchestrating intelligently across workflow steps by summarizing intermediate outputs and avoiding redundant context loading. It also introduces metrics for measuring token efficiency, such as tokens per successful task, latency per task, cost per task, retrieval precision, answer quality, and context utilization rate, and highlights common mistakes like over-focusing on prompt length and assuming large context windows solve underlying system design issues.