MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet

This post introduces MetaRoCE, a new RDMA transport protocol designed from the ground up for AI workloads on commodity Ethernet at million-GPU scale. Key technical contributions include native out-of-order delivery, native multipathing with packet-by-packet spraying across first-class paths, loss tolerance by design (eliminating PFC and pause frames), a combined sender-driven ECN-based and receiver-driven fair-share rate hint congestion control, topology independence by relying only on ECN marking and ECMP, and unified connections at scale by separating ordered streams from bandwidth. The post details practical implementation on AMD Pensando NICs, showing higher throughput and lower flow completion times than RoCEv2, graceful degradation under packet loss, and autonomous recovery from plane failures. The protocol specification, reference software implementation, and compliance test suite are being released through the Open Compute Project (OCP).