Arcadia: An end-to-end AI system performance simulator

What this post added

Introduced Arcadia, a unified end-to-end AI system performance simulator designed to model the compute, memory, and network performance of AI training clusters. Arcadia takes various inputs including AI system plans, network topology, workload distributions, and hardware specifications to provide insights into AI model and workload performance, enabling data-driven design decisions for AI clusters and infrastructure. It addresses challenges in optimizing large-scale, multi-layered AI systems by providing a common source of truth for performance analysis and joint optimization across application, network, and hardware levels.