# perf annotate

**`perf annotate` drills down to the instruction level**, showing source code or disassembly with per-instruction sample counts. This reveals exactly which line or instruction inside a hot function is causing stalls.

```bash
perf record -g ./program
perf annotate my_hot_function       # interactive browser
perf annotate --stdio -s my_hot_function  # text output
```

`perf annotate` requires debug symbols (`-g` flag during compilation) to map machine instructions back to source lines. Without debug info, it shows disassembly instead.

The interactive view displays source code (or disassembly if symbols are missing) with a column showing the percentage of samples on each line. Lines where the CPU spent time appear highlighted. This instantly shows which line is the bottleneck.

**Example output (simplified):**

```
Samples: 10K of event 'cycles:ppp'
Function: matrix_multiply

  0.00 |     for (i = 0; i < N; i++)
  5.23 |       for (j = 0; j < N; j++)
 94.77 |         C[i*N+j] += A[i*N+k] * B[k*N+j];
  0.00 |       }
```

The line doing the matrix multiply got 94.77% of samples—it's the hotspot. The loop overhead is negligible (0.00%).

**Disassembly view:**

```bash
perf annotate --no-source -s my_function   # show disassembly without source
```

If debug symbols aren't available or you want to see the actual CPU instructions, this shows disassembly with sample counts per instruction. Each `movq`, `add`, `cmp` is annotated with its percentage.

**Filtering:**

```bash
perf annotate --dsos libmpi.so -s MPI_Send   # annotate function in specific library
```

When profiling large programs with many libraries, filter to the library and function you're interested in.

**CPU stalls:** Look for patterns in the disassembly. A line with high samples followed by low samples on related instructions often indicates a cache miss or pipeline stall—the instruction is waiting for data from memory. This is a sign to optimize memory access patterns.

`perf annotate` is the deepest level of profiling. Use it after narrowing down to a specific function with `perf report`. Looking at millions of instruction samples is overwhelming; focus on the top 1-3 functions first.