TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

Multi-GPU Kernel Generation Benchmark

1 post · Jun 2026

What this trail covers

Together AI is developing and evaluating benchmarks for multi-GPU kernel generation, such as ParallelKernelBench (PKB), to assess the capabilities of frontier LLMs in writing production-ready CUDA kernels for distributed AI workloads. This involves creating a diverse set of problems derived from real codebases and evaluating models on correctness, speedup, and communication efficiency across various parallelism strategies.

All Together AI trails
  1. ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)This post introduces ParallelKernelBench (PKB), a new benchmark and evaluation framework designed to test LLMs' ability to generate multi-GPU CUDA kernels. PKB includes 87 problems sourced from real-world distributed AI systems (e.g., Megatron-LM, DeepSpeed, TensorRT-LLM) that require replacing PyTorch + NCCL implementations with optimized CUDA kernels for direct inter-GPU communication. The evaluation revealed that current frontier LLMs struggle significantly with multi-GPU kernel generation, with less than a third of problems solved correctly and fewer than a quarter of those outperforming naive baselines. The analysis highlights that LLMs fail due to difficulties in reasoning about rank coordination, data partitioning, and collective ordering, and they underutilize specialized communication mechanisms like TMA and NVLS. The benchmark is designed to evolve with next-generation hardware architectures.