Table of Contents

TFLOPS

TFLOPS is FLOPS measured in units of $10^{12}$ (teraflops) floating-point operations per second. This is the scale at which modern GPUs and small clusters are usually quoted: a single high-end datacenter GPU can exceed tens of TFLOPS in FP64, and well over a hundred TFLOPS in lower precision formats used for machine learning.

Why GPUs live in this range

A GPU reaches TFLOPS-scale throughput by trading single-core complexity for massive parallelism: thousands of simpler floating-point lanes running in lockstep across many streaming multiprocessors, rather than a few dozen complex cores each optimized for single-thread performance. Where a CPU core's GFLOPS comes from a handful of wide SIMD units, a GPU's TFLOPS comes from replicating comparatively narrow arithmetic units thousands of times over, which only pays off for workloads with enough independent parallel work to keep all those lanes busy, exactly the kind of workload CUDA is written to target.

Precision spread on modern GPUs

The gap between precision tiers is often larger on GPUs than on CPUs, because dedicated tensor-core-style hardware exists specifically for lower-precision matrix math. It's common for a single GPU's headline numbers to span roughly an order of magnitude between FP64 and FP16/bfloat16 tensor throughput, since the low-precision units are physically separate hardware from the general FP64 units, not just the same units running faster. Quoting a GPU's “TFLOPS” without precision is therefore close to meaningless; the number can differ by 10x or more for the same chip depending on which mode is meant.

Beyond TFLOPS

Clusters and supercomputers scale past single-GPU TFLOPS into petaflops ($10^{15}$) and exaflops ($10^{18}$) by combining many nodes over an Interconnect, at which point achieved throughput on real workloads is governed as much by communication overhead between nodes as by any single node's peak arithmetic rate.