Token-count-based Batching: Faster, Cheaper Embedding Inference for Queries

Introduced token-count-based batching for embedding inference, optimizing for short queries by batching based on total token count rather than request count or time windows. This approach leverages padding removal in inference engines (vLLM, SGLang) to align GPU work with actual token count. A Redis-backed queue with Lua scripting was implemented for atomic batching, resulting in a 50% reduction in GPU inference latency and a 3x reduction in GPU usage.