TechPrufer
Get started
Uber
Uber logoUber · Feature trail

LLM Training Infrastructure and Optimization

2 posts · Apr 2026

What this trail covers

Uber has developed a robust in-house LLM training infrastructure, Michelangelo, leveraging open-source tools like PyTorch, Ray, Hugging Face, and DeepSpeed. This platform supports both full-parameter fine-tuning and parameter-efficient methods like LoRA and QLoRA. The system has been optimized for throughput and Model Flops Utilization (MFU) through techniques such as DeepSpeed ZeRO-stage-3 CPU Optimizer Offload and flash attention, demonstrating scalability and efficiency on NVIDIA A100 and H10. This post details optimizations to Petastorm for high-throughput and reproducible GPU training, significantly improving GPU utilization to over 60%, reducing end-to-end training time by 86%, and slashing compute costs by nearly 80%. The optimizations include pushing data transformations to worker threads, implementing local disk caching with quota management, and ensuring deterministic data loading by caching row groups and adding a final shuffle step.

All Uber trails
  1. Open Source and In-House: How Uber Optimizes LLM TrainingThis post details Uber's in-house LLM training infrastructure, Michelangelo. It describes the hardware (NVIDIA A100/H100 GPUs), orchestration (Kubernetes, Ray), and training stack (PyTorch, DeepSpeed, Hugging Face). Key contributions include the distributed training pipeline architecture and performance optimizations. Specifically, it quantifies the benefits of DeepSpeed ZeRO-stage-3 CPU Optimizer Offload and flash attention on throughput and MFU for models like Llama 2 70B and 7B, analyzing whether training is compute-bound or memory-bound and noting network utilization.
  2. Accelerating Deep Learning: How Uber Optimized Petastorm for High-Throughput and Reproducible GPU TrainingThis post details optimizations to Petastorm for high-throughput and reproducible GPU training, significantly improving GPU utilization to over 60%, reducing end-to-end training time from 22 hours to 3 hours, and slashing compute costs by nearly 80%. The optimizations include pushing data transformations from PyArrow to NumPy to worker threads, implementing local disk caching using diskcache.FanoutCache with quota management, and ensuring deterministic data loading by caching row groups (not batches) and adding a final shuffle step in memory. It also addresses thread pool stability issues by implementing robust sentinel logic and stricter HDFS timeouts.