InfiniBand and Inter-Node Networking

A single H100 GPU delivers roughly 3.35 TFLOPS of FP64 throughput but connects to the rest of the cluster through a wire rated at perhaps 200 Gbps. When you are synchronising gradients across thousands of such cards, the arithmetic units often idle, waiting for bytes to arrive. Inter-node networking is not a footnote to the accelerator story; for many workloads it is the binding constraint.

What InfiniBand actually is

InfiniBand (IB) is a switched-fabric interconnect standard defined by the InfiniBand Trade Association. Unlike Ethernet, which was designed for bursty packet traffic with loose latency guarantees, IB was engineered from the start for consistent sub-microsecond latency and deterministic throughput.

Each generation roughly doubles the per-4x-link signalling rate:

The adapter latency drops from about 5 microseconds for early SDR hardware to under 0.6 microseconds for HDR. That sub-microsecond latency is not marketing; it matters for small-message collectives where synchronisation round trips dominate.

The defining feature is RDMA (Remote Direct Memory Access). With RDMA, a NIC can read from or write to a remote node's pinned memory without involving the remote CPU. The initiating CPU posts a work request; the hardware completes the transfer; a completion queue entry signals done. CPU overhead per transfer collapses from O(bytes) to O(1) for large payloads. This is why IB and its Ethernet cousin RoCE (RDMA over Converged Ethernet) dominate AI training clusters: gradient tensors can be moved peer-to-peer at wire speed without paying kernel overhead on every packet.

Fat-tree topology and why it matters

The physical switching architecture determines whether bandwidth compounds or contends.

A fat-tree builds a multi-root tree where each layer of switches has more upward-facing ports than the layer below it, guaranteeing full bisection bandwidth: any communication pattern between equal-sized halves of the cluster is served at full line rate simultaneously. In a two-layer fat-tree, leaf switches connect directly to servers; spine switches connect leaf switches to one another. The DeepSeek Fire-Flyer cluster, for example, uses a two-layer fat-tree with Mellanox QM8700 switches (40 ports at 200 Gbps each), and found this was significantly cheaper than the three-layer topology typical of larger DGX-based deployments while still providing adequate bandwidth for their workloads.

The alternative, cheaper topology is multi-rail with oversubscription: each server has multiple NICs attached to different switches, but total uplink bandwidth is smaller than total downlink bandwidth. This is fine if you can co-schedule communicating nodes on the same leaf pod. When they span pods, bandwidth is reduced and latency spikes.

A practical design choice: multi-rail with eight 200 Gbps NICs per server (as used in the MegaScale deployment at ByteDance) provides both redundancy and aggregate bandwidth without requiring a three-layer fabric, provided ECMP (Equal-Cost Multi-Path) hashing is tuned to avoid hot-spot collisions.