Performance model for MPI is $\alpha + \beta n$, where $\alpha$ is per-message latency and $\beta$ is per-byte time. Latency dominates small messages; bandwidth dominates large ones. Fewer larger messages are faster than many small ones.
// bad: N messages, each paying alpha for (int i = 0; i < N; i++) MPI_Send(&vals[i], 1, MPI_DOUBLE, dest, 0, MPI_COMM_WORLD); // good: one message, alpha paid once MPI_Send(vals, N, MPI_DOUBLE, dest, 0, MPI_COMM_WORLD);
This model also explains two MPI-specific behaviors. Small messages (typically below a few kilobytes, though the threshold is implementation-defined) use the eager protocol: the sender copies the data into a pre-allocated buffer and returns immediately; the receiver picks it up later. Large messages use the rendezvous protocol: sender and receiver handshake first, then data moves directly between their buffers with no intermediate copy. The handshake is why MPI_Send can block on large messages even when no explicit synchronisation is requested. It is also why buffering-based deadlocks often appear only at scale: small test data fits in the eager buffer and the code runs fine; production-size data triggers rendezvous and the code hangs.