numactl
numactl controls NUMA (Non-Uniform Memory Access) policy on multi-socket machines. Each CPU socket has its own attached memory. Cross-socket memory access is 2x slower than local access. Pin a process to CPU cores and memory on the same NUMA node to maintain locality—critical for HPC performance.
Multi-socket machines are not one uniform pool of RAM. Each socket has a memory controller and DIMMs; accessing memory from another socket crosses the inter-socket interconnect. The OS scheduler tries to keep threads and memory local automatically, but for HPC where every percentage of bandwidth matters, explicit binding is needed.
$ numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 node 0 size: 64000 MB node 1 cpus: 8 9 10 11 12 13 14 15 node 1 size: 64000 MB node distances: node 0 1 0: 10 21 1: 21 10
Distance 21 vs 10 means cross-socket access costs roughly 2x the latency. Bind with numactl --cpunodebind=0 --membind=0 ./program to keep everything on node 0.