Meta AI has released MetaRoCE, a new RDMA transport protocol designed specifically for AI workloads running on standard Ethernet hardware.
The specification, a reference software implementation, and a compliance test suite are available through the Open Compute Project (OCP). Release is scheduled for the 2026 OCP Global Summit, likely in October 2026.
The bottleneck is the network
Training and serving frontier models is now a networking problem as much as a compute problem. Collective operations like all-reduce and all-to-all synchronize thousands of accelerators during training, and the slowest transfer sets the pace for the entire job. Even small amounts of network friction directly strand significant compute capacity.
Meta has scaled clusters to hundreds of thousands of GPUs across multiple data centers and regions. At that size the network sits in the critical path of every training step. Standard RoCE is the constraint. It expects the network to deliver every frame in order, leans on PFC, and discourages the packet spraying that provides performance in multiplane and large-scale networks.
MetaRoCE inverts that: intelligence moves to the endpoint, and the network decomposes into many fine-grained logical paths, each with its own real-time telemetry — per-path RTT, ECN state, and utilization. This builds directly on Meta’s 2024 RoCE-at-scale work and its broader infrastructure evolution.
Six design decisions
- Out-of-order delivery is the default: Packets are sprayed across many paths and arrive out of order by design. Every packet carries its own destination, so data is written straight to its final memory location as it lands — no reorder buffer, no head-of-line blocking. Sends carry the match to a posted receive buffer, so a Send lands correctly even when messages ahead of it have not arrived.
- Multipathing is native: Each path carries a distinct UDP source port as its ECMP entropy, which the NIC can change at any time to move traffic off a bad route. Because each path keeps its own window and round-trip estimate, the transport can tell congestion from failure and rebalance explicitly.
- Loss tolerance replaces losslessness: MetaRoCE treats the fabric as lossy — no PFC, no pause frames. A gap in a path’s 256-bit selective acknowledgment bitvector is evidence of loss rather than reordering, so it triggers retransmission of exactly the missing packet, on the path that lost it.
- Congestion control runs from both ends: Sender-driven ECN-based AIMD is combined with receiver-driven fair-share rate hints. In every acknowledgment the receiver returns the share of inbound bandwidth it allocated to that sender, so senders approach the right speed directly rather than searching for it. Incast resolves in one or two round trips.
- Topology independence: MetaRoCE asks the fabric for two things every switch already has: ECN marking and ECMP. It does not require packet trimming, in-network telemetry, credit-based flow control, or switch-side spraying — which means it also runs over vendor clouds whose configuration you don’t control.
- Connection state stops exploding: Traditional RDMA gets more ordering or bandwidth by opening more queue pairs — dozens per node pair — each with a congestion window blind to the rest. MetaRoCE separates the two: one connection carries many independent ordered streams above and many paths below, under one congestion controller.
The numbers
Meta implemented MetaRoCE on AMD Pensando programmable NICs. On a 64-node AMD GPU cluster running RCCL collectives, it was compared directly against RoCEv2 across all-reduce and all-to-all, delivering higher throughput and lower flow completion times.
The resilience result is the core statement: MetaRoCE maintains ~86% throughput at 1% packet loss and continues delivering useful bandwidth even at 10% loss rates, converging gracefully rather than collapsing. Multiplane validation across 4-plane and 8-plane topologies with up to 4,000 concurrent connections confirmed throughput scales linearly with plane count, and simulated plane failures showed traffic redistributing without application involvement or operator intervention.
Open by design
MetaRoCE extends the multi-vendor philosophy that OCP’s Ethernet Scalable Unified Network (ESUN) initiative established for the fabric into the transport layer. Three artifacts ship: the full spec via OCP, a compliance suite that lets vendors prove their implementations match, and libsoftmetaroce as the authoritative behavioral model for silicon development. Meta has proven it on AMD Pensando hardware, with additional implementations underway from other vendors.
What it means
For the people building these systems, the change is simple: stop fighting for perfect order. The new protocol accepts that the network will drop packets and that traffic will split across many routes. It handles the recovery and the routing logic inside the network interface card, allowing data to write directly to memory without waiting for a queue to clear. This removes the head-of-line blocking that usually stalls large-scale training jobs.
