Token efficiency: getting more signal into the context window

This post details the challenges of token inefficiency in LLMs, leading to degraded reasoning and specific failure modes (context poisoning, distraction, confusion, clash) due to the 'lost in the middle' effect and attention budget limitations. It explains how low-signal tokens (boilerplate, duplicates, irrelevant tool schemas) consume valuable context window space. The post proposes solutions for high-signal token selection: reranking (using cross-encoders to improve Mean Reciprocal Rank), hybrid search with metadata filtering (combining vector and keyword search), context compression (e.g., LLMLingua for token perplexity scoring), and pruning/ordering of context. It also introduces semantic caching as a method to avoid unnecessary LLM calls by indexing responses with query embeddings.