Token-count-based Batching: Faster, Cheaper Embedding Inference for Queries

What this post added

This post introduces and details the implementation of token-count-based batching for embedding inference. It explains the inefficiencies of sequential processing and traditional batching methods for short queries, highlights the importance of padding removal in modern inference engines, proposes token-count-based batching as a solution, discusses optimal batch size determination through latency profiling, and describes a Redis-based queue implementation for atomic batch acquisition. The results show significant improvements in latency, GPU utilization, and throughput.