How we built BEI: high-throughput embedding, reranker, and classifier inference

This post details the development of Baseten Embedding Inference (BEI), an optimized inference runtime built on TensorRT-LLM. It addresses the challenges of high-throughput corpus processing and low-latency real-time querying for modern, larger embedding models. Key technical contributions include leveraging TensorRT-LLM for performance, implementing sequence packing for efficient batching without OOM errors, and optimizing for both high throughput (up to 2.05x improvement with FP8 quantization) and low latency for individual requests. The post also discusses infrastructure considerations like queueing, backpressure, and network overhead.