A read() call on a file descriptor blocks: it doesn't return until the kernel has data to hand back. That is convenient but ties up the thread for the duration of the wait. POSIX offers aio_read() as an alternative: issue the request, do other work, and check completion later with aio_suspend(). MPI has the same split. MPI_Send and MPI_Recv are blocking: MPI_Send does not return until the send buffer is safe to reuse, and MPI_Recv does not return until the receive buffer holds the data. MPI_Isend and MPI_Irecv are the non-blocking counterparts — they return immediately with a request handle, and MPI_Wait blocks until the transfer is complete.
MPI_Request req; int x = 42; MPI_Isend(&x, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &req); do_other_work(); // overlap computation with communication MPI_Wait(&req, MPI_STATUS_IGNORE); // x must not change before this point
MPI_Test checks whether a request is complete without blocking, returning a flag. MPI_Waitall and MPI_Testall handle arrays of requests, which is common when a process has posted non-blocking sends to multiple neighbors. Non-blocking communication is the primary tool for hiding communication latency behind useful computation.