Designing low-latency AI agents through reranker optimization | Decagon

What this post added

This post details the optimization of the reranking stage within the RAG pipeline for low-latency AI agents. It explains the difference between list-wise and point-wise rerankers, highlighting the latency benefits of point-wise reranking due to shorter sequence lengths and predictable per-call costs. The core technical contribution is the implementation of efficient batching using attention masking, which allows multiple (query, document) pairs to be scored in a single batched forward pass by creating a block-diagonal attention pattern. The post also discusses finding the optimal batch size for minimizing end-to-end latency, empirically showing batch size 2 as optimal for their system.