numactl OpenMP

Combining numactl with OpenMP ensures threads land on intended cores with local memory. OpenMP's affinity mechanism (OMP_PLACES, OMP_PROC_BIND) specifies which threads run on which cores; numactl enforces it system-wide and controls memory allocation.

Basic setup:

export OMP_NUM_THREADS=8
export OMP_PLACES="{0:4},{4:4}"       # threads 0-3 on core 0, threads 4-7 on core 4
export OMP_PROC_BIND=close
numactl --cpunodebind=0 --membind=0 ./app

OMP_PLACES specifies thread placement (cores 0-3 and 4-7). OMP_PROC_BIND=close keeps threads close to their initial placement. numactl enforces that the entire process stays on node 0.

More complex affinity:

export OMP_PLACES="{0,1,2,3},{4,5,6,7}"  # explicit core lists
export OMP_PROC_BIND=spread              # spread threads across places

OMP_PLACES can list cores explicitly (comma-separated) or use ranges. OMP_PROC_BIND=spread distributes threads evenly across places; OMP_PROC_BIND=close clusters them.

Per-NUMA-node binding:

export OMP_NUM_THREADS=8
export OMP_PLACES="{0:4},{4:4}"
export OMP_PROC_BIND=close
numactl --cpunodebind=0-1 --membind=0-1 --interleave=0-1 ./app

Threads 0-3 run on cores 0-3 (node 0), threads 4-7 on cores 4-7 (node 1). Memory is interleaved across both nodes. Each thread's private data (stack, thread-local storage) lives on its home node.

Nested OpenMP with NUMA:

export OMP_NUM_THREADS=4
export OMP_PLACES="{0,1,2,3}"
export OMP_PROC_BIND=close
numactl --cpunodebind=0 --membind=0 ./app

Bind the entire process to node 0, then let OpenMP do fine-grained affinity within that node.

Debugging affinity:

export OMP_DISPLAY_ENV=true
numactl --cpunodebind=0 --membind=0 ./app

OMP_DISPLAY_ENV=true prints OpenMP settings at startup, showing which cores threads are pinned to.

Performance tip: Ensure OMP_NUM_THREADS matches the cores you're pinning to. Pinning to 4 cores but setting OMP_NUM_THREADS=8 causes contention and defeats the purpose.

Note: OpenMP affinity is a hint; numactl is a guarantee. numactl overrides OS scheduling, ensuring the process stays pinned even if OpenMP's affinity isn't perfectly configured.