TechPrufer
Get started
Exa
Exa logoExa · Feature trail

Real-time Embedding Serving

1 post · Feb 2024

What this trail covers

Exa has developed a high-throughput system for serving real-time embeddings, crucial for features like Exa Highlights. This system has evolved from Python to Rust, significantly improving performance and addressing bottlenecks related to CPU-GPU communication and GIL contention. The current iteration focuses on efficient parallel processing across multiple GPUs and optimizing CUDA operations to prevent fragmentation and out-of-memory errors.

All Exa trails
  1. Scaling Our Server for Semantic Search Technology HighlightsThis post details the migration of Exa Highlights' embedding serving pipeline from Python to Rust to overcome GIL contention and improve CPU-GPU parallelism. It highlights the challenges of measuring CUDA operations, identifying OS scheduling as a bottleneck, and the benefits of Rust's native parallelism. The post also describes a critical issue with CUDA out-of-memory errors due to thread-unsafe CUDA calls in a parallel iterator context, which was resolved by refactoring inference and deallocation into separate serial passes.