Nsight is NVIDIA's suite of profiling and debugging tools for CUDA and GPU-accelerated applications. It is really two separate tools sharing a brand name: Nsight Systems, a timeline profiler that shows how CPU and GPU work overlaps across an entire application run, and Nsight Compute, a kernel-level profiler that dissects a single CUDA kernel launch down to instruction throughput, memory bandwidth, and occupancy.
perf and valgrind were built for CPU execution and have no visibility into what happens once work is dispatched onto a GPU; the GPU's execution is asynchronous relative to the CPU and runs its own scheduler across thousands of concurrent threads. Nsight exists because profiling a heterogeneous CPU+GPU application needs a tool that understands CUDA streams, kernel launches, and memory transfers as first-class events, not just CPU-side function calls.
The two tools answer different questions and are used in sequence. Nsight Systems answers “where is the overall time going, and is the GPU actually staying busy or waiting on the CPU,” which is the first thing to check on any new workload. Nsight Compute answers “why is this specific kernel slow,” and is reached for only after Systems has identified which kernel is worth that level of scrutiny.
Nsight Systems and Nsight Compute ship with the CUDA Toolkit, or can be installed standalone from NVIDIA's developer site. On Debian and Ubuntu, once the NVIDIA CUDA APT repository is configured:
sudo apt install nsight-systems nsight-compute
Both tools require an NVIDIA GPU and driver on the target machine; profiling a remote GPU node typically means running the CLI collection step on the remote machine over ssh and copying the resulting report file back to view in the GUI locally.
nsys --version ncu --version
Nsight Systems wraps a whole program invocation and records a timeline of everything that happened, CPU threads, CUDA API calls, kernel execution, memory copies:
nsys profile -o report ./my_cuda_app
This produces report.nsys-rep, opened in the Nsight Systems GUI to see a timeline view: CPU activity on one row, each CUDA stream on its own row below, so gaps where the GPU sits idle waiting on the CPU (or vice versa) are immediately visible as literal empty space in the timeline.
Nsight Compute profiles one kernel invocation at a time in much greater depth:
ncu -o kernel_report ./my_cuda_app
By default this profiles every kernel launch in the program, which is slow; narrowing to a specific kernel by name and launch count is standard practice once Systems has identified the target:
ncu --kernel-name my_hot_kernel --launch-count 1 -o kernel_report ./my_cuda_app
The core insight Nsight Systems gives is whether the GPU is actually the bottleneck at all. A common surprise on a first profiling pass is a timeline where the GPU rows show frequent gaps, meaning the application is CPU-bound (waiting on data preparation, disk I/O, or single-threaded host code) rather than GPU-bound, and no amount of kernel optimisation will help until the CPU-side stall is fixed.
nsys profile --trace=cuda,nvtx,osrt -o report ./app
--trace=nvtx picks up NVTX range annotations the application itself can emit to label logical phases (like “data load” vs “training step”) directly in the timeline, which is far more useful than trying to infer program phases from raw kernel names alone.
Nsight Compute reports metrics like achieved occupancy (fraction of the GPU's thread-scheduling capacity actually in use), memory throughput as a percentage of the device's theoretical peak bandwidth, and compute throughput as a percentage of peak FLOPs. A kernel with low occupancy and low memory throughput both is usually latency-bound rather than bandwidth- or compute-bound, often from too few threads in flight to hide instruction latency.
ncu --set full --kernel-name my_hot_kernel -o full_report ./app
--set full collects the complete metric set, including the roofline-adjacent breakdown of whether the kernel is compute-bound or memory-bound, at the cost of running the kernel many times (once per metric group) to collect it all.
Nsight Compute can generate a roofline plot for a profiled kernel, plotting its achieved performance against the theoretical peak the GPU can deliver at that kernel's arithmetic intensity (FLOPs per byte moved from memory). A kernel plotted well below the roofline ceiling for its intensity has room to improve; whether that means restructuring memory access patterns or increasing arithmetic intensity per byte loaded depends on which side of the roofline's knee the kernel sits on.
The typical HPC pattern is to collect a report on the remote GPU node over ssh with the CLI tools (nsys/ncu), then transfer the resulting .nsys-rep or .ncu-rep file back to a local machine to open in the GUI, since the GUI itself doesn't need GPU access, only the collection step does.
ssh gpu-node "nsys profile -o /tmp/report ./app" scp gpu-node:/tmp/report.nsys-rep .