numactl Performance

NUMA effects vary by workload. A memory-bandwidth-bound HPC code suffers significantly from cross-socket access; a CPU-bound code with small working set might not.

Measuring NUMA impact: Run the same workload with and without binding:

# Without binding (default OS policy)
time ./app < input
 
# With binding to node 0
time numactl --cpunodebind=0 --membind=0 ./app < input
 
# With binding to node 1
time numactl --cpunodebind=1 --membind=1 ./app < input

Compare runtimes. If binding to node 0 vs node 1 shows similar times, NUMA effects are weak. If they differ by 5-20%, NUMA locality matters.

Bandwidth-limited code: Matrix multiplication, stencil codes, and other data-intensive workloads often see 10-20% speedup from NUMA binding. These codes move gigabytes of data per second; cross-socket bandwidth is limited, so local access wins.

Compute-limited code: Kernels that do lots of math with small working sets (e.g., prime number search) see little benefit from binding. The working set fits in cache regardless of NUMA node.

Scaling across nodes: On multi-node clusters (HPC systems), NUMA binding matters per node, but MPI handles inter-node communication. Focus on intra-node binding—keep threads and memory on the same node within each MPI rank.

Pinning trade-offs:

Binding helps:

• Memory bandwidth stays local
• Less contention on inter-socket interconnect
• Predictable performance

Binding hurts:

• Load imbalance (all threads on one node while another idles)
• Hyperthreading contention (pinning fewer cores than available)
• Fragmented memory (if you pin too tightly)

For best performance, benchmark your workload. Don't assume binding helps—measure.

Multi-socket vs single-socket: On machines with 4+ sockets, NUMA effects compound. Cross-socket distance grows, and a thread on socket 3 accessing memory on socket 0 pays even more latency. Binding becomes more critical.

Profiling with perf: Measure cache misses and bandwidth:

perf stat -e LLC-loads,LLC-load-misses,LLC-stores,LLC-store-misses \
    numactl --cpunodebind=0 --membind=0 ./app
 
perf stat -e LLC-loads,LLC-load-misses,LLC-stores,LLC-store-misses ./app

Compare the two runs. If binding reduces LLC misses, NUMA locality is helping.

Optimal binding strategy:

1. Measure baseline performance (no binding)
2. Try single-node binding
3. Try multi-node binding if single node is too small
4. Use numastat to verify memory placement
5. Use perf to measure cache behavior
6. Compare runtimes

If binding doesn't help, the workload isn't NUMA-sensitive. Leave it alone—OS defaults are fine.