numactl Diagnosing

numastat reports where a process's memory actually lives on each NUMA node. Use it to verify that bindings are working as intended.

numastat -p $(pgrep myprogram)

Example output:

Per-node process memory usage (in MBs)
PID             Node 0          Node 1           Total
1234567         1024            512              1536

This process has 1024 MB on node 0 and 512 MB on node 1—not perfectly local (you'd want all 1536 MB on one node).

Interpreting results:

Perfect binding (all memory on node 0):

PID  Node0  Node1  Total
123  1536   0      1536

Scattered memory (problem):

PID  Node0  Node1  Total
123  768    768    1536

Scattered memory usually means the binding was applied after the process started. Binding must happen at process launch before the first allocation.

Real-time memory monitoring:

watch -n 1 'numastat -p $(pgrep myprogram)'

Updates every second. Watch the numbers change as the program allocates and deallocates.

System-wide memory distribution:

numastat

Shows total memory usage on each node and OS overhead. Useful for understanding machine-wide NUMA balance.

Debugging failed bindings:

If binding doesn't seem to work, check:

1. Is numactl working? Check if the process is actually running with the binding:

taskset -p -c $(pgrep myprogram)    # show CPU affinity

If it shows all CPUs instead of your pinned subset, the binding failed.

1. Did you bind at startup? Bindings must happen before the first allocation:

# Correct
numactl --cpunodebind=0 --membind=0 ./app
 
# Wrong
./app &
numactl --cpunodebind=0 -- kill %1 && exec numactl --cpunodebind=0 ./app

1. Is the workload actually running on the pinned node? If threads are pinned but the memory is scattered, the workload may be allocating from other nodes (e.g., library code, OS buffers). Check with numastat.
2. Hyperthreading interference? On systems with hyperthreading, pinning to only even cores might help:

numactl --physcpubind=0,2,4,6 --membind=0 ./app

Combining numastat with perf: Profile memory behavior:

perf record -e LLC-load-misses ./app
perf report
numastat -p $(pgrep myprogram)  # after the run

High LLC misses combined with scattered memory placement confirms NUMA effects.