Token-Count-Based Batching for Embedding Inference

Introduced token-count-based batching to optimize embedding model inference for short requests. This approach leverages padding removal in inference engines like vLLM and SGLang to process sequences based on total token count rather than fixed batch sizes or time windows. A Redis-backed queue system with Lua scripting was implemented to atomically batch requests by token count, achieving up to a 50% reduction in GPU inference latency and a 3x reduction in GPU usage. MongoDB is integrating Voyage AI's embedding and reranking models into Atlas to deliver a truly native experience, simplifying developer workflows and enhancing accuracy, performance, and cost efficiency. This includes exploring multi-modal retrieval, instruction-tuned retrieval, domain-specific models, and easier model updates.