Hybrid MPI+OpenMP combines MPI for inter-node communication with OpenMP for intra-node parallelism. One MPI rank per node with OpenMP threads is the dominant HPC model.

When threads are involved, MPI must be initialised with MPI_Init_thread instead of MPI_Init to declare the required level of thread safety:

int provided;
MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &provided);
if (provided < MPI_THREAD_FUNNELED) {
    fprintf(stderr, "insufficient MPI thread support\n");
    MPI_Abort(MPI_COMM_WORLD, 1);
}

The four thread-safety levels are:

• MPI_THREAD_SINGLE — only one thread will execute; equivalent to MPI_Init
• MPI_THREAD_FUNNELED — multiple threads exist but only the main thread makes MPI calls; the most common level for MPI+OpenMP
• MPI_THREAD_SERIALIZED — multiple threads make MPI calls but not concurrently; the application serialises them
• MPI_THREAD_MULTIPLE — multiple threads call MPI concurrently; requires a thread-safe MPI build and has higher overhead

With MPI_THREAD_FUNNELED, all MPI calls must happen on the master thread, either outside parallel regions or inside one guarded with #pragma omp master. The typical structure is to post non-blocking communication on the master thread, enter a parallel region to compute the interior while halos travel, then wait for communication before computing the boundary.

while (!converged) {
    MPI_Startall(nreqs, reqs);           // post halo exchange on master thread
    #pragma omp parallel for
    for (int i = interior_lo; i < interior_hi; i++)
        update(i);                        // interior computation overlaps with comms
    MPI_Waitall(nreqs, reqs, MPI_STATUSES_IGNORE);
    #pragma omp parallel for
    for (int i = 0; i < halo_size; i++)
        update_halo(i);
}