MPI

MPI (Message Passing Interface) is a standard for distributed-memory parallel programming in C, C++, and Fortran. Launch N independent copies of your program via mpirun -n N ./program; each gets its own address space and communicates by explicitly sending and receiving messages. Unlike OpenMP, processes don't share memory, but MPI scales from a laptop to clusters with thousands of nodes without code changes.

The execution model is SPMD (single program, multiple data): all processes start together and execute the same binary, but each plays a different role based on its rank (0 to N-1). Every MPI program must call MPI_Init first and MPI_Finalize last. Compile with mpicc (C) or mpicxx (C++).

// compile: mpicc -o ping ping.c
// run: mpirun -n 2 ./ping
// description: rank 0 sends a value to rank 1
 
#include <mpi.h>
#include <stdio.h>
 
int main(int argc, char **argv) {
    MPI_Init(&argc, &argv);
    int rank, value = 0;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
 
    if (rank == 0) {
        value = 42;
        MPI_Send(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
        printf("rank 0: sent %d\n", value);
    } else {
        MPI_Recv(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
        printf("rank 1: received %d\n", value);
    }
 
    MPI_Finalize();
    return 0;
}

Rank 0 blocks on MPI_Send until rank 1 reaches MPI_Recv. This blocking behaviour is fundamental—MPI communication is synchronous. From here, point-to-point communication is the natural next concept, then collectives.

1. Communicators
2. Point-to-point communication
3. Blocking and non-blocking
4. Send modes
5. Message ordering
6. Probing for messages
7. Deadlock
8. Performance model
9. Collectives
10. Reduction
11. Scatter and gather
12. Prefix reductions
13. Non-blocking collectives
14. Persistent communication
15. Derived datatypes
16. Virtual topologies
17. Process groups
18. Communicator duplication
19. One-sided communication
20. Shared memory windows
21. Parallel I/O
22. Time measurement
23. Hybrid MPI+OpenMP
24. Overview