TechPrufer
Get started
Baseten
Baseten logoBaseten · Feature trail

Baseten Performance Client for Embedding Throughput

1 post · Jun 2025

What this trail covers

Baseten has developed and open-sourced the Baseten Performance Client, a Python library built with Rust, designed to significantly improve embedding throughput for high-volume tasks. This client addresses Python's Global Interpreter Lock (GIL) limitations by offloading network-bound operations to a Rust-based Tokio runtime, enabling true parallel request execution and maximizing multi-core CPU utilization. It offers up to 12x higher throughput and lower latencies compared to standard clients, supporting both synchronous and asynchronous usage, and general batch requests for modalities beyond embeddings.

All Baseten trails
  1. Your client code matters: 12x higher embedding throughput with Python and RustIntroduced the Baseten Performance Client, an open-source Python library built with Rust, to achieve up to 12x higher embedding throughput by releasing the Python GIL during network-bound tasks. This allows for true parallel request execution using a Rust Tokio runtime, leading to significantly lower latencies under heavy loads and efficient multi-core CPU utilization. The client supports both synchronous and asynchronous APIs and general batch requests for various inference tasks.