Scaling Our Server for Semantic Search Technology Highlights

This post details the optimization of the Exa Highlights real-time embedding service, migrating from Python to Rust to achieve 4x throughput by overcoming GIL contention and improving parallelism for CPU-bound text pre-processing and GPU-bound model inference. It also addresses challenges with CUDA memory fragmentation and thread safety when using parallel iterators with GPUs.