# Interconnect
An **interconnect** is the network fabric that moves data between compute nodes, or between sockets within a node, in a parallel system. Whatever the scale, the interconnect exists to answer the same question: once a computation is split across more than one processor, how does data get from where it was produced to where it's needed. At small scale that's an on-chip bus or ring between cores or sockets; at cluster scale it's a dedicated fabric like [[infiniband]] connecting whole machines.

## Latency vs bandwidth

An interconnect is characterized by two largely independent numbers: **latency**, the fixed time to get any message across regardless of size, and **bandwidth**, the sustained rate of large transfers once they're underway. A workload dominated by many small messages (fine-grained synchronization, small MPI messages) is latency-bound, and no amount of extra bandwidth helps it. A workload dominated by large bulk transfers (checkpoint writes, big collective operations) is bandwidth-bound, and latency barely matters once the transfer is large enough to amortize it. HPC interconnect design (and application tuning) largely comes down to figuring out which regime a given communication pattern falls into and optimizing for that one.

## Topology

How nodes are wired together shapes both cost and worst-case latency. A **fat tree** connects nodes through a hierarchy of switches with increasing bandwidth toward the root, giving good bisection bandwidth at the cost of more switches and cabling than a simpler topology. A **torus** or **mesh** connects each node directly to its geometric neighbors, which scales cheaply (wiring grows linearly with node count) but means far-apart nodes communicate through many hops, so worst-case latency grows with system size rather than staying flat. Real supercomputers pick a topology based on their expected communication pattern: a workload that's mostly nearest-neighbor exchange (a physical simulation on a spatial grid) fits a torus naturally, while an all-to-all collective favors a fat tree's flatter worst case.

## RDMA as the software-visible payoff

The interconnect's hardware topology mostly stays invisible to application code, which talks to it through a library like [[mpi]] or directly through [[rdma]]. RDMA is what lets that library move data between nodes without invoking the OS kernel or copying through an intermediate buffer on every transfer, which is where a fast physical interconnect actually translates into an application seeing low latency, rather than the fabric's speed being wasted on software overhead layered on top of it.