Cascading retrieval with multi-vector representations: balancing efficiency and effectiveness

Introduced ConstBERT, a novel fixed-size multi-vector representation model that significantly reduces storage overhead (by 50% or more) and improves query latency compared to traditional variable-size multi-vector models like ColBERT. This makes multi-vector retrieval more scalable and practical for integration into retrieval pipelines as an intermediate step between first-stage retrieval and expensive cross-encoder reranking. The post details the challenges of traditional multi-vector approaches at scale (memory, compute, variable document lengths) and how ConstBERT's fixed-size representation addresses these by optimizing indexing, cache-friendliness, and batch processing. It also illustrates a cascading retrieval pipeline incorporating ConstBERT.