Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI

This post details engineering lessons learned from generating 12 trillion synthetic tokens for AI model pre-training. It covers the transition from a Slurm/Kubernetes setup to a unified Ray, KubeRay, and vLLM pipeline on EKS. Key bottlenecks addressed include optimizing S3 metadata fetches (reducing 11 days to 2 hours), implementing right-sized partitions and checkpointing for recoverable GPU failures, orchestrating CPU and GPU across clusters, and tuning vLLM for approximately 40% increased inference throughput. The talk also touches upon the 'BeyondWeb' synthetic data recipe based on seeded rephrasing.