perf is a Linux kernel profiling tool that reads hardware performance counters—built-in CPU registers that count events like cycles, instructions, cache misses, and branches. Install it via your package manager and configure permissions to allow non-root profiling.

sudo apt install linux-perf                    # Debian/Ubuntu
sudo dnf install perf                          # Fedora/RHEL

By default, non-root users cannot access hardware counters due to security restrictions. Check the current restriction level:

cat /proc/sys/kernel/perf_event_paranoid

A value of 2 (or higher) restricts most events to root. For development work, set it to 0:

echo 0 | sudo tee /proc/sys/kernel/perf_event_paranoid

Make this permanent by adding kernel.perf_event_paranoid = 0 to /etc/sysctl.d/99-perf.conf, then run sudo sysctl -p to reload.

Compilation flags matter for profiling. Compile with -g to include DWARF debug symbols, which allow perf to show function names, source line numbers, and call stacks. Optimization flags like -O2 and -O3 should be kept on—profiling unoptimized code tells you nothing useful about performance.

gcc -O3 -g -o program program.c     # correct: optimized with debug info
gcc -O0 -g -o program program.c     # wrong for profiling: no optimization

The -g flag adds debug symbols; the -O3 flag applies optimizations. These are compatible and should both be used for accurate profiling of HPC code.

Basic workflow: Start with perf stat for a high-level overview of what the hardware is doing (cycles, instructions, cache behavior). If the overview shows something interesting, use perf record to sample where in the code the problem is. Finally, use perf report or perf annotate to drill down into specific functions or instructions.