# numactl Diagnosing

**`numastat`** reports where a process's memory actually lives on each NUMA node. Use it to verify that bindings are working as intended.

```bash
numastat -p $(pgrep myprogram)
```

Example output:

```
Per-node process memory usage (in MBs)
PID             Node 0          Node 1           Total
1234567         1024            512              1536
```

This process has 1024 MB on node 0 and 512 MB on node 1—not perfectly local (you'd want all 1536 MB on one node).

**Interpreting results:**

Perfect binding (all memory on node 0):
```
PID  Node0  Node1  Total
123  1536   0      1536
```

Scattered memory (problem):
```
PID  Node0  Node1  Total
123  768    768    1536
```

Scattered memory usually means the binding was applied after the process started. Binding must happen at process launch before the first allocation.

**Real-time memory monitoring:**

```bash
watch -n 1 'numastat -p $(pgrep myprogram)'
```

Updates every second. Watch the numbers change as the program allocates and deallocates.

**System-wide memory distribution:**

```bash
numastat
```

Shows total memory usage on each node and OS overhead. Useful for understanding machine-wide NUMA balance.

**Debugging failed bindings:**

If binding doesn't seem to work, check:

1. **Is numactl working?** Check if the process is actually running with the binding:

```bash
taskset -p -c $(pgrep myprogram)    # show CPU affinity
```

If it shows all CPUs instead of your pinned subset, the binding failed.

2. **Did you bind at startup?** Bindings must happen before the first allocation:

```bash
# Correct
numactl --cpunodebind=0 --membind=0 ./app

# Wrong
./app &
numactl --cpunodebind=0 -- kill %1 && exec numactl --cpunodebind=0 ./app
```

3. **Is the workload actually running on the pinned node?** If threads are pinned but the memory is scattered, the workload may be allocating from other nodes (e.g., library code, OS buffers). Check with `numastat`.

4. **Hyperthreading interference?** On systems with hyperthreading, pinning to only even cores might help:

```bash
numactl --physcpubind=0,2,4,6 --membind=0 ./app
```

**Combining numastat with perf:** Profile memory behavior:

```bash
perf record -e LLC-load-misses ./app
perf report
numastat -p $(pgrep myprogram)  # after the run
```

High LLC misses combined with scattered memory placement confirms NUMA effects.