'I paid for the whole GPU, I am going to use the whole GPU': A high-level guide to GPU utilization

This post introduces a framework for understanding GPU utilization, breaking it down into GPU Allocation Utilization, GPU Kernel Utilization, and Model FLOP/s Utilization. It details strategies for improving each of these, with a focus on neural network inference. For GPU Allocation Utilization, it highlights the importance of fast, automatic allocation and spin-up, and how Modal addresses this with its multi-cloud GPU fleet and custom container stack. For GPU Kernel Utilization, it discusses reducing host overhead and improving GPU throughput through techniques like CUDA Graphs and batching. The post also touches on Model FLOP/s Utilization by discussing the theoretical arithmetic bandwidth of GPUs.