# RDMA
**RDMA** (Remote Direct Memory Access) lets one machine read or write another machine's memory directly over an [[interconnect]], without involving the remote CPU or its operating system in the transfer. A conventional network send has to traverse the kernel's networking stack on both ends and typically copies data at least once into a socket buffer; RDMA skips both, letting the network adapter move data straight between the two applications' memory.

```
Conventional TCP:   app -> kernel copy -> NIC -> wire -> NIC -> kernel copy -> app
RDMA:               app memory -> NIC -> wire -> NIC -> remote app memory (no kernel, no copy)
```

## Why this matters for latency

Every kernel transition and every memory copy adds fixed overhead that dominates at small message sizes, exactly the regime latency-sensitive HPC and distributed-training communication lives in. By removing both, RDMA gets latency down to close to what the physical fabric alone would take, rather than paying software overhead on top of it. This is the mechanism [[infiniband]] is built around natively, though RDMA also exists over Ethernet as **RoCE** (RDMA over Converged Ethernet), trading some of InfiniBand's native efficiency for the ability to reuse existing Ethernet infrastructure.

## Registration and one-sided operations

Before memory can be touched by RDMA, it has to be **registered** with the network adapter, pinning it in physical memory (so it can't be paged out mid-transfer) and giving the adapter permission to access it. Once registered, RDMA supports **one-sided** operations: a `RDMA_WRITE` or `RDMA_READ` completes without the remote side's software running any code at all to service it, in contrast to a normal socket `recv()`, which requires the remote application to actively call into its own networking stack to accept the data. One-sided operations are what let a sender push (or pull) data at very low latency even if the remote CPU is busy doing something completely unrelated at that moment.

## Where it shows up

MPI implementations use RDMA underneath collective and point-to-point calls on any interconnect that supports it. GPUDirect RDMA extends the same idea one step further, letting the network adapter read and write GPU memory directly, bypassing host RAM entirely, which is a major part of why multi-node GPU training scales as well as it does on RDMA-capable fabrics.