perf annotate drills down to the instruction level, showing source code or disassembly with per-instruction sample counts. This reveals exactly which line or instruction inside a hot function is causing stalls.
perf record -g ./program perf annotate my_hot_function # interactive browser perf annotate --stdio -s my_hot_function # text output
perf annotate requires debug symbols (-g flag during compilation) to map machine instructions back to source lines. Without debug info, it shows disassembly instead.
The interactive view displays source code (or disassembly if symbols are missing) with a column showing the percentage of samples on each line. Lines where the CPU spent time appear highlighted. This instantly shows which line is the bottleneck.
Example output (simplified):
Samples: 10K of event 'cycles:ppp' Function: matrix_multiply 0.00 | for (i = 0; i < N; i++) 5.23 | for (j = 0; j < N; j++) 94.77 | C[i*N+j] += A[i*N+k] * B[k*N+j]; 0.00 | }
The line doing the matrix multiply got 94.77% of samples—it's the hotspot. The loop overhead is negligible (0.00%).
Disassembly view:
perf annotate --no-source -s my_function # show disassembly without source
If debug symbols aren't available or you want to see the actual CPU instructions, this shows disassembly with sample counts per instruction. Each movq, add, cmp is annotated with its percentage.
Filtering:
perf annotate --dsos libmpi.so -s MPI_Send # annotate function in specific library
When profiling large programs with many libraries, filter to the library and function you're interested in.
CPU stalls: Look for patterns in the disassembly. A line with high samples followed by low samples on related instructions often indicates a cache miss or pipeline stall—the instruction is waiting for data from memory. This is a sign to optimize memory access patterns.
perf annotate is the deepest level of profiling. Use it after narrowing down to a specific function with perf report. Looking at millions of instruction samples is overwhelming; focus on the top 1-3 functions first.