TechPrufer
Sign inGet started

Blogs›MongoDB›Token-Count-Based Batching for Embedding Inference

Token-Count-Based Batching for Embedding Inference

Token-Count-Based Batching for Embedding Inference

1
posts
2025

Introduced token-count-based batching to optimize embedding model inference for short queries. This approach leverages padding removal in inference engines like vLLM and SGLang to process sequences based on total token count rather than fixed batch sizes or time windows. A Redis-backed queue system with Lua scripting was implemented to atomically batch requests by token count, achieving up to a 50% reduction in GPU inference latency and a 3x reduction in GPU usage.

2025

Token-count-based Batching: Faster, Cheaper Embedding Inference for Queries

12/18/2025

This post introduces and details the implementation of token-count-based batching for embedding inference. It explains the inefficiencies of sequential processing and traditional batching methods for short queries, highlights the importance of padding removal in modern inference engines, proposes token-count-based batching as a solution, discusses optimal batch size determination through latency profiling, and describes a Redis-based queue implementation for atomic batch acquisition. The results show significant improvements in latency, GPU utilization, and throughput.