TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

Hardware Utilization and Software-Hardware Co-design

3 posts · Dec 2025 – Sep 2026

What this trail covers

Together AI continues to push the boundaries of AI hardware utilization and software-hardware co-design, exemplified by their work on the NVIDIA Vera Rubin NVL72 platform. This post details the integration of ThunderKittens, a GEMM kernel, with the new ISA and features of Vera Rubin, achieving over 22 PFLOPS in NVFP4 GEMMs. The team has optimized the kernel by leveraging doubled K-step instructions, increased tensor memory, oversized shared memory, B-side collectors, and early A-release mechanisms. This post also argues that current hardware is significantly underutilized (~20% MFU for training, single digits for inference) and that better software-hardware co-design and innovations like FP4 training will unlock the next order of magnitude in performance, challenging the notion that AI is hitting a hardware wall.

All Together AI trails
  1. To Infinity and Beyond: ThunderKittens Now on NVIDIA Vera Rubin NVL72!This post details the integration of the ThunderKittens GEMM kernel with the NVIDIA Vera Rubin NVL72 platform. It highlights new features such as doubled K-step instructions for tensor cores, increased tensor memory (576 columns), oversized shared memory (328 KiB), B-side collector functionality for operand reuse, and early A-release commit signals. The authors demonstrate how these features are leveraged to optimize the kernel, achieving over 22 PFLOPS in NVFP4 GEMMs by addressing bottlenecks in data supply to the tensor cores and improving data reuse.
  2. ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)Introduces ParallelKernelBench (PKB), a benchmark and evaluation framework for multi-GPU kernel generation. PKB consists of 87 problems derived from real codebases, focusing on replacing PyTorch + NCCL with CUDA kernels that move data directly over NVLink. The post presents evaluation results for frontier coding models, demonstrating significant performance gaps and failures in generating correct and optimized multi-GPU kernels. It analyzes the reasons for these failures, attributing them to the combinatorial expansion of the design space, changes in the performance model (interconnect bottlenecks), and the critical new design choice of data movement between GPUs. The post also notes that generated kernels utilize a narrow set of communication mechanisms, with specialized ones like TMA and NVLS being largely absent.
  3. Inside the Together AI kernels teamThis post details the work of the Together AI kernels team, focusing on their development of ThunderKittens and Together Megakernel. ThunderKittens is an abstraction library for NVIDIA tensor cores, reducing CUDA code complexity and enabling rapid adaptation to new hardware like Blackwell GPUs. Together Megakernel is a production implementation that hand-optimizes entire models into a single kernel for specific workloads, achieving significant latency reductions for real-time voice agents. The post highlights the symbiotic relationship between academic research (Stanford, UCSD, Princeton) and productionization at Together AI, emphasizing the importance of kernel optimization for AI-native infrastructure and customer deployments.
  4. Research POV: Yes, AGI Can Happen – A Computational PerspectiveThis post argues that current AI hardware is significantly underutilized, with training runs achieving only ~20% Mean FLOP Utilization (MFU) and inference utilization in the single digits. It posits that better software-hardware co-design and innovations like FP4 training can unlock substantial performance gains, challenging the idea that AI is hitting a hardware wall. It also highlights that future compute generations have not yet been fully utilized and that current models are already transforming complex workflows like GPU kernel generation.