MPI rank binding assigns each rank to a NUMA node, keeping the rank's threads and memory local. Most important on shared-memory HPC systems where multiple MPI ranks run on the same node.

Basic MPI binding:

# 2 ranks, each on its own NUMA node
mpirun -n 2 numactl --cpunodebind=\$((OMPI_COMM_WORLD_RANK % 2)) \
    --membind=\$((OMPI_COMM_WORLD_RANK % 2)) ./app

OMPI_COMM_WORLD_RANK is OpenMPI's rank environment variable. The % operator distributes ranks round-robin: rank 0 → node 0, rank 1 → node 1, rank 2 → node 0, rank 3 → node 1, etc.

Simpler syntax (MPICH/OpenMPI):

mpirun -n 2 -bind-to hwthread:list=0,4 ./app

This is MPI-implementation-specific. OpenMPI supports direct binding; MPICH uses –bind-to. Check your MPI's documentation.

Multi-threaded MPI (hybrid parallelism):

export OMP_NUM_THREADS=4
mpirun -n 2 \
    bash -c 'numactl --cpunodebind=$((OMPI_COMM_WORLD_RANK % 2)) \
        --membind=$((OMPI_COMM_WORLD_RANK % 2)) \
        --physcpubind=$((OMPI_COMM_WORLD_RANK * 4 % 8))-$((OMPI_COMM_WORLD_RANK * 4 + 3 % 8)) ./app'

Rank 0 gets threads 0-3 on node 0; rank 1 gets threads 4-7 on node 1. Each rank's OpenMP threads stay on the rank's home node.

Binding verification:

mpirun -n 2 bash -c 'echo "Rank $OMPI_COMM_WORLD_RANK: $(taskset -p -c $$)"'

Prints the CPU affinity for each rank. Verify that rank 0 is pinned to cores on node 0, rank 1 to node 1, etc.

With OpenMP affinity:

export OMP_NUM_THREADS=4
export OMP_PLACES="{0,1,2,3},{4,5,6,7}"
export OMP_PROC_BIND=close
 
mpirun -n 2 bash -c 'numactl --cpunodebind=$((OMPI_COMM_WORLD_RANK % 2)) \
    --membind=$((OMPI_COMM_WORLD_RANK % 2)) ./app'

Rank 0 pins to node 0; its OpenMP threads pin to cores 0-3. Rank 1 pins to node 1; its threads pin to cores 4-7. Full locality control.

Oversubscription: If you have more MPI ranks than NUMA nodes, multiple ranks share one node. This can cause memory and cache contention. Usually better to reduce rank count or use inter-node communication (true MPI) if possible.

Profiling MPI + numactl:

mpirun -n 2 bash -c 'numactl --cpunodebind=$((OMPI_COMM_WORLD_RANK % 2)) \
    --membind=$((OMPI_COMM_WORLD_RANK % 2)) \
    perf stat -e LLC-loads,LLC-load-misses,instructions,cycles ./app'

Run perf inside each rank to measure cache behavior per rank. Verify that bindings are reducing cross-socket misses.

Note: MPI's inter-rank communication (over the network or shared-memory) is much slower than intra-rank thread communication. Binding focuses on intra-node locality. Inter-node topology is handled by MPI's communication patterns.