Lessons › AI infrastructure networking
RDMA and RoCE: making Ethernet lossless
▶ Start the interactive lesson
The customer chose Ethernet for their GPU fabric. On day one, a test job shows "RDMA timeouts" and terrible speed. The switches show packet drops during bursts. Sam: "Normal Ethernet is allowed to drop packets when it's busy. TCP copes. RDMA doesn't. We have to teach this network not to drop."
RDMA: skip the middleman
Normally, sending data means the CPU and operating system copy it several times before it leaves the server. That adds delay.
RoCE: RDMA over Ethernet
RDMA was born on InfiniBand. RoCE (say "rocky") runs RDMA over normal Ethernet. RoCE v2 puts it inside UDP/IP, so it can be routed across a leaf-spine fabric.
PFC: "pause, my buffer is full"
PFC (priority flow control) lets a switch tell the sender: "stop sending on this traffic class for a moment, my buffer is nearly full." The sender pauses instead of the switch dropping.
ECN: "slow down a bit"
ECN (explicit congestion notification) lets a switch mark packets when its queue starts filling, before it's full.
Key points
- RDMA writes straight into remote memory: low latency, low CPU.
- RoCE v2 = RDMA over routable Ethernet. It needs a lossless fabric.
- ECN marks early so senders slow down smoothly. PFC pauses a traffic class as the last-resort safety net.
- Put RDMA in its own class with the right QoS, and watch for PFC storms.
Words you'll learn
RDMA · RoCE v2 · Lossless Ethernet · PFC (priority flow control, 802.1Qbb) · ECN · DCQCN · PFC storm / head-of-line blocking · RNIC / SmartNIC / SuperNIC