From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model

This post details how to significantly improve GPU utilization during multimodal AI model training by addressing data pipeline bottlenecks. It demonstrates a transition from 15% to 90% GPU utilization by optimizing data loading, decoding, and resizing. Key techniques include using asyncio and Ray actors for concurrency, prefetching data to ensure it's ready for the trainer, and employing Ray's object store for zero-copy tensor transfers. At scale, a network card bottleneck was resolved by distributing workers across nodes and implementing zero-copy reads, leading to a 50% throughput increase. The talk highlights that multimodal training is often CPU- and IO-bound, not GPU-bound, and discusses the trade-offs between asyncio, threads, and processes around the GIL.