InfiniBand is a switched-fabric Interconnect used across most large HPC clusters and many AI training clusters, designed from the start around low latency and RDMA rather than being adapted from a general-purpose networking standard like Ethernet. Where Ethernet's design goal is broad compatibility with everything from a home router to a datacenter, InfiniBand's design goal is narrower and more specific: get a message from one node's memory to another node's memory with the least possible latency and CPU involvement.

InfiniBand's host channel adapters (HCAs) support RDMA as a first-class operation, not a bolted-on extension: a remote memory read or write is issued directly to the HCA and completed by the fabric without the receiving node's CPU or OS being interrupted at all. This is why InfiniBand latencies (commonly under 1-2 microseconds for the fabric hop) are well below what an equivalent Ethernet-based RDMA setup (RoCE, RDMA over Converged Ethernet) typically achieves, since InfiniBand's whole stack, not just the wire protocol, was built around this goal from day one.

Large InfiniBand deployments are almost always wired as a fat tree (see Interconnect), giving every pair of nodes comparable worst-case latency and high aggregate bisection bandwidth regardless of which two nodes happen to be communicating. Adaptive routing lets the fabric spread traffic across multiple equal-cost paths dynamically, avoiding hotspots that a static routing scheme would create if many flows happened to collide on the same physical link.

Applications rarely talk to InfiniBand hardware directly. MPI implementations (OpenMPI, MVAPICH) use InfiniBand's verbs API underneath standard MPI_Send/MPI_Recv calls, so a program written against MPI gets InfiniBand's low latency without any InfiniBand-specific code. NCCL, the collective communication library behind most multi-GPU deep learning training, does the same for GPU-to-GPU transfers across nodes, using InfiniBand (often combined with GPUDirect RDMA, which lets the HCA read GPU memory directly without an intermediate copy through host RAM) as its default high-performance transport.