Accelerating Deep Learning: How Uber Optimized Petastorm for High-Throughput and Reproducible GPU Training

This post details optimizations to Petastorm for high-throughput and reproducible GPU training, significantly improving GPU utilization to over 60%, reducing end-to-end training time from 22 hours to 3 hours, and slashing compute costs by nearly 80%. The optimizations include pushing data transformations from PyArrow to NumPy to worker threads, implementing local disk caching using diskcache.FanoutCache with quota management, and ensuring deterministic data loading by caching row groups (not batches) and adding a final shuffle step in memory. It also addresses thread pool stability issues by implementing robust sentinel logic and stricter HDFS timeouts.