Beyond prompt engineering: the real drivers of token efficiency in enterprise AI

What this post added

This post details how Glean's AI platform achieves token efficiency by focusing on retrieval quality, context selection, and orchestration design, rather than solely on prompt engineering. It introduces a four-layer framework: retrieving better and less, passing only relevant evidence through intelligent chunking and passage selection, structuring prompts for control, and orchestrating intelligently across steps by summarizing intermediate outputs and avoiding redundant context loading. The post emphasizes measuring token efficiency through metrics like tokens per successful task, latency per task, cost per task, retrieval precision, answer quality, and context utilization rate, highlighting common mistakes like over-reliance on large context windows or optimizing for cost alone.