Lessons › AI infrastructure networking
InfiniBand basics for network engineers
▶ Start the interactive lesson
Your next customer has an InfiniBand cluster. You open the switch and... no VLANs, no IP routing tables like you're used to, no BGP. There's something called a "subnet manager" and addresses called "LIDs". Sam: "Relax. The ideas are familiar: switches, paths, addresses, congestion. It just does…
What InfiniBand is
InfiniBand (IB) is a network built for supercomputers and AI clusters. It was designed from the start for RDMA, very low latency and no packet loss.
Lossless by design: credits
Ethernet sends and hopes there's room. InfiniBand uses credit-based flow control: a sender only sends when the next hop has said "I have buffer space for you".
The subnet manager
An IB network (called a subnet) is run by a subnet manager (SM). It discovers every switch and adapter, gives each one an address (a LID), and calculates the paths.
InfiniBand or Ethernet?
InfiniBand: lossless by design, very low latency, mature for AI, but specialist skills and mostly one main vendor.
Key points
- InfiniBand is a separate network technology built for RDMA, low latency and no loss.
- Credit-based flow control: send only when the next hop has room, so no congestion drops.
- A subnet manager discovers devices, assigns LIDs and calculates paths. Run a standby.
- IB vs Ethernet + RoCE: both power big AI clusters. Knowing both is valuable.
Words you'll learn
HCA (host channel adapter) · HDR 200G, NDR 400G, XDR 800G · Credit-based flow control · Subnet manager (SM) · LID (local identifier) · OpenSM / UFM · IB subnet