**This is an old revision of the document!**
Table of Contents
Parallel computing
Parallel computing is a type of software engineering where you use parallelism (more threads, more cores, more computers) to increase the amount of computation power against a given task. There are essentially three spheres of parallel computing: 1. CPU parallelism (OpenMP), 2. distributed parallelism (MPI), and 3. GPU parallelism (CUDA).
Parallel computing is related to performance engineering. This makes sense, because usually increasing parallelism increases performance, but this is not always so. Not every piece of code can be parallelized and sometimes the overhead of creating threads can overweight the potential speedup.
Generally speaking, the potential speed up you'd get by parallelizing increasing $N$ cores is governed by Amdahl's law $$S(\text{N-cores}) = \frac{1}{(1 - P) + \frac{P}{N}}$$
Arguably the most important