Blogs›Together AI›Hardware Utilization and Software-Hardware Co-design

Hardware Utilization and Software-Hardware Co-design

posts

2025–2026

This post introduces the concept of hardware utilization and software-hardware co-design as a key driver for future AI progress, challenging the notion of hitting a hardware wall. It highlights that current AI systems are significantly underutilizing existing hardware (e.g., ~20% MFU for training, single digits for inference) and that future compute generations are yet to be fully leveraged. The post also points to the utility of current models in complex tasks like GPU kernel generation, suggesting a new benchmark, ParallelKernelBench (PKB), for evaluating LLM capabilities in generating multi-GPU kernels. PKB reveals that current frontier LLMs struggle with multi-GPU kernel generation, often failing to compile or produce incorrect results, and rarely outperform naive baselines. This highlights a gap in LLM reasoning about rank coordination, data partitioning, and collective ordering, as well as a lack of proficiency with specialized communication mechanisms like TMA and NVLS.

2026

ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)

6/23/2026

Introduces ParallelKernelBench (PKB), a benchmark and evaluation framework for multi-GPU kernel generation. PKB consists of 87 problems derived from real codebases, focusing on replacing PyTorch + NCCL with CUDA kernels that move data directly over NVLink. The post presents evaluation results for frontier coding models, demonstrating significant performance gaps and failures in generating correct and optimized multi-GPU kernels. It analyzes the reasons for these failures, attributing them to the combinatorial expansion of the design space, changes in the performance model (interconnect bottlenecks), and the critical new design choice of data movement between GPUs. The post also notes that generated kernels utilize a narrow set of communication mechanisms, with specialized ones like TMA and NVLS being largely absent.

2025

Research POV: Yes, AGI Can Happen – A Computational Perspective

12/17/2025

The post argues that current AI systems are significantly underutilizing existing hardware, with Mean FLOP Utilization (MFU) often around 20% for training and single digits for inference. It posits that better software-hardware co-design and innovations like FP4 training can unlock substantial performance gains, challenging the idea that AI progress is solely limited by hardware constraints. It also notes that future compute generations have not yet been fully integrated into AI development, and that current models already demonstrate significant utility in complex tasks.