Differences

This shows you the differences between two versions of the page.

--- parallel-computing [June 10, 2026 at 21:45] – Ivan Janevski
+++ parallel-computing [June 15, 2026 at 15:47] (current) – Ivan Janevski
@@ Line 1: / Line 1: @@
 # Parallel computing
-**Parallel computing** is a type of software engineering where you use parallelism (more threads, more cores, more computers) to increase the amount of computation power against a given task. There are essentially three spheres of parallel computing: 1. CPU parallelism ([[openmp|OpenMP]]), 2. distributed parallelism ([[mpi|MPI]]), and 3. GPU parallelism ([[cuda|CUDA]]).
+**Parallel computing** is a style of programming where a computation is broken into parts that run simultaneously across multiple processors, cores, or machines. The motivation is straightforward: a single core has a clock speed ceiling, and modern CPUs gain performance by adding more cores rather than running each core faster. To take advantage of that, programs have to be written with parallelism in mind.
-Parallel computing is related to performance engineering. This makes sense, because usually increasing parallelism increases performance, but this is not always so. Not every piece of code can be parallelized and sometimes the overhead of creating threads can overweight the potential speedup.
+Not every program benefits equally. [[amdahls-law|Amdahl's law]] shows that the sequential fraction of a program — the part that cannot be parallelized — sets a hard ceiling on speedup regardless of how many cores you add. [[gustafsons-law|Gustafson's law]] is the more optimistic counterpart: if you scale the problem size alongside the hardware, speedup grows linearly. In practice, HPC workloads follow Gustafson's regime — you buy more nodes to solve a bigger problem, not just to solve the same one faster.
-Generally speaking, the potential speed up you'd get by parallelizing increasing $N$ cores is governed by Amdahl's law
+## Map of parallel computing
-$$S(\text{N-cores}) = \frac{1}{(1 - P) + \frac{P}{N}}$$
-Arguably the most important
-## List of parallel computing concepts
+ - [[introduction-to-parallel-computing]]
+ - [[saxpy]]
+ - [[synchronization-primitve]]
+   - [[sync-semaphore]]
+   - [[sync-mutex]]
+   - [[sync-monitor]]
+   - [[sync-linda]]
+   - [[sync-csp]]
+   - [[sync-mbox]]
+ - [[numbers-every-programmer-should-know]]
+ - [[amdahls-law]]
+ - [[gustafsons-law]]
+ - [[cache]]
+   - [[l1-cache]]
+   - [[l2-cache]]
+   - [[l3-cache]]
+   - [[cache-coherence]]
+     - [[cache-snoopy-protocols]]
+        - [[wti]]
+        - [[msi]]
+        - [[mesi]]
+        - [[moesi]]
+        - [[dragon]]
+        - [[firefly]]
+     - [[cache-directory-protocols]]
+ - [[cuda]]
  - [[openmp]]
  - [[mpi]]
- - [[cuda]]
+ - [[queuing-theory]]
+ - [[numa]]
+ - [[false-sharing]]
+ - [[aba-problem]]
+ - [[trace-monoid]]
+ - [[hazard-pointer]]
+ - [[cache-coherence]]
+ - [[roofline-model]]
+ - [[numa]]
+ - [[embarrassingly-parallel]]
+ - [[fork-join-model]]
+ - [[rcu]]
+ - [[lock]]
+ - [[lock-convoy]]
+ - [[lock-contention]]
+ - [[spinlock]]
+ - [[smp]]
+ - [[flops]]
+   - [[gflops]]
+   - [[tflops]]
+ - [[atomics]]
+   - [[atomic-ops]]
+     - [[atomic-load-store]]
+       - [[atomic-load]]
+       - [[atomic-store]]
+     - [[atomic-read-modify-write]]
+       - [[atomic-test-and-set]]
+       - [[atomic-exchange]] or [[xchg]]
+       - [[atomic-compare-and-swap]] or [[cas]]
+     - [[atomic-fetch-modify]]
+       - [[atomic-fetch-arithmetic]]
+         - [[atomic-fetch-inc]]
+         - [[atomic-fetch-dec]]
+         - [[atomic-fetch-add]]
+         - [[atomic-fetch-sub]]
+       - [[atomic-fetch-bitwise]]
+         - [[atomic-fetch-and]]
+         - [[atomic-fetch-or]]
+         - [[atomic-fetch-xor]]
+       - [[atomic-fetch-reduce]]
+         - [[atomic-fetch-min]]
+         - [[atomic-fetch-max]]
+     - [[atomic-flag]]
+     - [[atomic-wait-notify]]
+       - [[atomic-wait]]
+       - [[atomic-notify-one]]
+       - [[atomic-notify-all]]
+   - [[lock-free-queue]]
+   - [[memory-order]]
+     - [[memory-order-relaxed]]
+     - [[memory-order-consume]]
+     - [[memory-order-acquire]]
+     - [[memory-order-acq-rel]]
+     - [[memory-order-seq-cst]]
+ - [[treiber-stack]]
+ - [[michael-scott-queue]]
+ - [[bespoke-algorithm]]
+ - [[interconnect]]
+   - [[infiniband]]
+   - [[rdma]]