Open Source and In-House: How Uber Optimizes LLM Training

What this post added

This post details Uber's in-house LLM training infrastructure and pipeline, focusing on the Michelangelo platform. It describes the hardware (A100, H100 GPUs), orchestration (Kubernetes, Ray), and training stack (PyTorch, DeepSpeed, Hugging Face Transformers, NCCL). The distributed training pipeline is explained, covering multi-host/GPU communication, data preparation, model training, and checkpoint management. The post also presents results on training state-of-the-art LLMs and explores throughput/MFU optimization strategies like CPU offload and flash attention, analyzing their impact on different model sizes (Llama 2 70B vs. 7B) and hardware (A100 vs. H100).