TechPrufer
Get started
Qdrant
Qdrant logoQdrant · Feature trail

Cross-encoder Inference Optimization

1 post · Sep 2026

What this trail covers

Qdrant is developing and optimizing cross-encoder inference capabilities, focusing on performance improvements through Rust implementation and efficient use of ONNX Runtime. This includes benchmarking against Python libraries to demonstrate performance gains in latency and model load times, particularly for larger models.

All Qdrant trails
  1. Oxidizing Cross-Encoders - QdrantThis post details the "oxidation" of cross-encoder inference into a Rust library, cross-encode-rs. It explains the underlying concepts of cross-encoders and ONNX Runtime, outlines the Rust implementation for tokenization, batching, and inference, and presents benchmarks comparing its performance against Python libraries (fastembed and sentence-transformers) using ONNX Runtime. The benchmarks show that cross-encode-rs offers competitive or improved inference latency, especially for larger models, and comparable model load times.