Token efficiency: getting more signal into the context window

What this post added

This post elaborates on the challenges of context window limitations in LLMs, introducing the concepts of 'context rot' and specific failure modes (poisoning, distraction, confusion, clash). It details the sources of low-signal tokens and proposes techniques for improving token efficiency, including reranking, hybrid search with metadata filtering, context compression, and strategic pruning/ordering of context. It also introduces semantic caching as a method to reduce token expenditure.