Large clusters for small models — Daniel Svonava, Superlinked

This post details how Superlinked addresses the bottleneck of serving many small models by inverting the routing model to a pull-based system, doubling cluster throughput, and using autoresearch to tune model configurations, including an 80-cent LoRA that improved retrieval on German legal text by 18%. It highlights the challenges of untuned open-source serving tools, stale worker state in top-down routers leading to low utilization, and the velocity killer of frequent LoRA/fine-tune conversations. The solution involves a gateway annotating requests, dropping them into a shared queue, and workers pulling tasks to form their own batches. A Rust sidecar abstracts adapters over three runtimes.