Boosting multimodal inference performance by >10% with a single Python dictionary | Modal Blog

What this post added

This post details an optimization in SGLang's scheduler to improve multimodal inference performance. By profiling the scheduler with `py-spy`, a bottleneck was identified in the `process_input_requests` function, specifically within calls to `torch.UntypedStorage._new_shared_cuda`. This was caused by repeatedly reopening shared CUDA IPC handles for tensors. The solution implemented a Python dictionary cache (`_pool_storage_cache`) to store and reuse these handles, significantly reducing host-side overhead. This optimization resulted in a ~16% increase in throughput and a ~10% reduction in mean end-to-end latency for multimodal workloads, and also improved decode latency by reducing overall scheduler contention.