Table of Contents

numactl Memory Policies

Memory allocation policies control where malloc, new, and page faults allocate memory. --membind is strict; --preferred is flexible; --interleave spreads memory across nodes.

--membind (strict binding): 

numactl --cpunodebind=0 --membind=0 ./app

All memory comes from node 0. If node 0 runs out, the allocation fails (even if other nodes have free memory). Use when you know the memory footprint and want to guarantee local access.

--preferred (fallback): 

numactl --cpunodebind=0 --preferred=0 ./app

Try to allocate from node 0, but fall back to other nodes if node 0 is full. Allocation never fails due to NUMA constraints. Useful for workloads of variable size or when you can tolerate some cross-socket access.

--interleave (round-robin): 

numactl --cpunodebind=0-1 --interleave=0,1 ./app
numactl --cpunodebind=all --interleave=all ./app

Spread memory pages round-robin across specified nodes. Each allocation alternates: first page from node 0, second from node 1, third from node 0, etc. Good for large shared data structures accessed uniformly by threads on all nodes. Bad for working sets that fit on one node—you pay cross-socket penalties unnecessarily.

Choosing a policy:

  1. Single-socket workload: --membind=N with --cpunodebind=N. Everything local, no cross-socket access.
  2. Multi-socket workload with private per-thread data: Bind threads to their home node's CPUs, use --preferred or --membind on that node. Each thread's stack and private data stay local.
  3. Shared data structure accessed by all nodes: Use --interleave=all to spread the structure evenly. Reduces hot spots on any one memory controller.
  4. Unknown or variable memory footprint: Use --preferred=N with --cpunodebind=N. Tries to keep memory local but doesn't fail if the workload grows large.

Example: MPI with per-rank binding: 

# Rank 0 on node 0, rank 1 on node 1
mpirun -n 2 bash -c 'numactl --cpunodebind=$((OMPI_COMM_WORLD_RANK % 2)) \
    --membind=$((OMPI_COMM_WORLD_RANK % 2)) ./app'

Each MPI rank binds to its assigned node. Rank 0 → node 0, rank 1 → node 1. Ensures locality within each rank.

Interleave vs non-interleaved: Interleave is a trade-off. It avoids memory controller hot spots and load imbalance but costs more cross-socket access if the working set could fit on one node. Benchmark both.