Designing low-latency AI agents through reranker optimization | Decagon

This post details how Decagon reduced reranking latency in their RAG pipeline by transitioning from list-wise to point-wise reranking. They implemented an efficient batching strategy using attention masking to score multiple (query, document) pairs in a single batched forward pass, preserving scoring semantics while amortizing model call overhead. Empirical results showed that a batch size of 2 yielded the best latency improvements. This optimization was achieved through structural insight and targeted changes rather than overhauling the underlying model.