NetWarren_

Lessons › AI infrastructure networking

InfiniBand basics for network engineers

11 min lesson · AI infrastructure networking · free

▶ Start the interactive lesson

Your next customer has an InfiniBand cluster. You open the switch and... no VLANs, no IP routing tables like you're used to, no BGP. There's something called a "subnet manager" and addresses called "LIDs". Sam: "Relax. The ideas are familiar: switches, paths, addresses, congestion. It just does…

What InfiniBand is

InfiniBand (IB) is a network built for supercomputers and AI clusters. It was designed from the start for RDMA, very low latency and no packet loss.

Lossless by design: credits

Ethernet sends and hopes there's room. InfiniBand uses credit-based flow control: a sender only sends when the next hop has said "I have buffer space for you".

The subnet manager

An IB network (called a subnet) is run by a subnet manager (SM). It discovers every switch and adapter, gives each one an address (a LID), and calculates the paths.

InfiniBand or Ethernet?

InfiniBand: lossless by design, very low latency, mature for AI, but specialist skills and mostly one main vendor.

Key points

Words you'll learn

HCA (host channel adapter) · HDR 200G, NDR 400G, XDR 800G · Credit-based flow control · Subnet manager (SM) · LID (local identifier) · OpenSM / UFM · IB subnet

The full lesson has more: animations, a hands-on exercise, 3 quiz questions, a practical scenario test and spaced review cards. Open it free →