Open Source and In-House: How Uber Optimizes LLM Training

This post details Uber's in-house LLM training infrastructure and pipeline, named Michelangelo. It covers the hardware (NVIDIA A100/H100 GPUs), orchestration (Kubernetes, Ray), and training stack (PyTorch, Hugging Face, DeepSpeed, NCCL). The distributed training pipeline is described, including communication, data preparation, model training, and checkpointing. The post also presents results on training state-of-the-art LLMs like Llama 2, focusing on throughput and Model Flops Utilization (MFU) optimization through techniques like DeepSpeed ZeRO-stage-3 CPU Offload and flash attention, and analyzes compute vs. memory bound scenarios for different model sizes.