Lessons › AI infrastructure networking
Why AI needs a special network
▶ Start the interactive lesson
A company has bought 512 GPUs to train its own AI model. They're expensive: each GPU server costs more than a house. Week one, the training job runs at half the expected speed. The GPUs are fine. The servers are fine. The problem is the network. Sam: "In AI clusters, the network isn't just…
GPUs work as one big team
AI models are too big to train on one GPU. The work is split across hundreds or thousands of GPUs, in many servers.
Why normal networks aren't enough
Normal data centre traffic is lots of small, random flows. A few dropped packets just get resent and nobody notices.
Two networks, not one
AI clusters usually have separate networks: a front-end network (users, storage, management, like a normal data centre) and a back-end or compute fabric just for GPU-to-GPU traffic.
Two ways to build the GPU fabric
InfiniBand: a network built for supercomputers. Lossless, very low latency, but its own technology and tools.
Key points
- Training splits work over many GPUs, which must share results (all-reduce) every step.
- The slowest link sets the pace for the whole job.
- AI fabrics need huge bandwidth, low predictable latency, and no loss, usually as a separate non-blocking back-end network.
- Built with InfiniBand or Ethernet with RoCE, both using RDMA.
Words you'll learn
Collective communication · All-reduce · Front-end network · Back-end / compute fabric · NVLink / scale-up network · Scale-out network · Tail latency matters · GPU cluster / AI factory / SuperPOD