ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)

What this post added

Introduces ParallelKernelBench (PKB), a benchmark and evaluation framework for multi-GPU kernel generation. PKB consists of 87 problems derived from real codebases, focusing on replacing PyTorch + NCCL with CUDA kernels that move data directly over NVLink. The post presents evaluation results for frontier coding models, demonstrating significant performance gaps and failures in generating correct and optimized multi-GPU kernels. It analyzes the reasons for these failures, attributing them to the combinatorial expansion of the design space, changes in the performance model (interconnect bottlenecks), and the critical new design choice of data movement between GPUs. The post also notes that generated kernels utilize a narrow set of communication mechanisms, with specialized ones like TMA and NVLS being largely absent.