NetWarren_

Lessons › AI infrastructure networking

RDMA and RoCE: making Ethernet lossless

13 min lesson · AI infrastructure networking · free

▶ Start the interactive lesson

The customer chose Ethernet for their GPU fabric. On day one, a test job shows "RDMA timeouts" and terrible speed. The switches show packet drops during bursts. Sam: "Normal Ethernet is allowed to drop packets when it's busy. TCP copes. RDMA doesn't. We have to teach this network not to drop."

RDMA: skip the middleman

Normally, sending data means the CPU and operating system copy it several times before it leaves the server. That adds delay.

RoCE: RDMA over Ethernet

RDMA was born on InfiniBand. RoCE (say "rocky") runs RDMA over normal Ethernet. RoCE v2 puts it inside UDP/IP, so it can be routed across a leaf-spine fabric.

PFC: "pause, my buffer is full"

PFC (priority flow control) lets a switch tell the sender: "stop sending on this traffic class for a moment, my buffer is nearly full." The sender pauses instead of the switch dropping.

ECN: "slow down a bit"

ECN (explicit congestion notification) lets a switch mark packets when its queue starts filling, before it's full.

Key points

Words you'll learn

RDMA · RoCE v2 · Lossless Ethernet · PFC (priority flow control, 802.1Qbb) · ECN · DCQCN · PFC storm / head-of-line blocking · RNIC / SmartNIC / SuperNIC

The full lesson has more: animations, a hands-on exercise, 3 quiz questions, a practical scenario test and spaced review cards. Open it free →