Ivan's wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
# perf Flame Graphs

**Flame graphs** visualize call-graph profiling data as stacked horizontal bars, where bar width represents CPU time spent in each function. Developed by Brendan Gregg, they're ideal for understanding which function call chains consume the most CPU.

```bash
perf record --call-graph dwarf -F 99 ./program
perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg
```

The `perf script` command exports samples in a text format. `stackcollapse-perf.pl` aggregates the data into stack traces (available from the FlameGraph repository on GitHub). `flamegraph.pl` converts stack traces into an SVG file.

**Reading a flame graph:** Each horizontal bar represents a function. The width of the bar is proportional to the percentage of total CPU time spent in that function (including its children). Functions are stacked vertically with their callers below and callees above. The widest bars at the top are the leaf functions where the CPU actually spent time.

For example, if you see `main` → `compute_kernel` → `matrix_multiply` as a tall stack, and `matrix_multiply` is wide, then `matrix_multiply` is your hotspot and you should optimize it.

**Interactive SVG:** The generated flame graph is an interactive SVG file. Click any bar to zoom into that subtree—the view expands to show only that function and its children. This helps drill down into specific call chains. Press Escape to unzoom.

**Advantages over `perf report`:**
1. Call chains are visible as complete stacks, not just individual functions.
2. Multi-level context is clear (seeing the full call path from main to a leaf function).
3. Wide/narrow bars immediately show which functions are expensive without needing to expand trees.
4. Interactive exploration makes it easy to navigate deep call stacks.

**MPI and library code:** Flame graphs are especially useful for profiling MPI programs where execution passes through many library layers:

```
application_function
  → MPI_Allreduce
    → ompi_coll_tuned_allreduce
      → mca_btl_tcp_send
        → syscall
```

`perf report` alone shows time in `mca_btl_tcp_send` with no context. The flame graph shows the complete chain—which application function initiated the MPI call—revealing the actual bottleneck in your code.

**Generating with sampling:** Use a reasonable sampling rate (`-F 99` for 99 Hz) to avoid excessive overhead. Higher rates (e.g., `-F 1000`) produce noisier data with more overhead; lower rates may miss small functions. 99 Hz is a good default.

**Filtering the data:** `perf script` can filter before flamegraph conversion:

```bash
perf script | grep myprogram | stackcollapse-perf.pl | flamegraph.pl > flame.svg
```

This focuses the graph on a specific binary or library, removing noise from other processes.

For large MPI jobs with many processes, flame graphs can become cluttered. Extracting profiles from individual ranks and creating separate flame graphs is often clearer than trying to visualize everything at once.