A cache is a small, fast memory that sits between the CPU and main memory (DRAM), holding copies of recently used data so that most accesses never have to pay DRAM's full latency. Main memory access takes hundreds of cycles; a well-tuned cache serves the same data in a handful of cycles. The whole idea rests on two empirical properties of real programs: temporal locality (a location accessed once is likely to be accessed again soon) and spatial locality (a location near a recently accessed one is likely to be accessed soon too).
Modern CPUs implement this as a hierarchy rather than a single cache. Each level trades capacity for speed: smaller and faster closer to the core, larger and slower closer to memory.
core -> [[l1-cache|L1]] (fastest, smallest, per-core)
-> [[l2-cache|L2]] (per-core or per-cluster)
-> [[l3-cache|L3]] (shared across cores, largest, slowest)
-> main memory (DRAM)
Caches don't move individual bytes between levels; they move fixed-size chunks called cache lines, typically 64 bytes on x86 and ARM. Loading one byte pulls in its entire line, which is what makes spatial locality pay off: iterating over an array sequentially only pays the miss cost once every 64 bytes, not once per element. This same granularity is also the root cause of False sharing, where two threads touching unrelated variables that happen to share a line contend with each other anyway.
A cache hit finds the requested line already resident; a cache miss has to fetch it from the next level down, stalling the core (or, on out-of-order CPUs, allowing other instructions to proceed while the miss is outstanding). Where a given memory address is allowed to live in the cache is governed by its associativity: a direct-mapped cache maps each address to exactly one slot, an N-way set-associative cache allows it to live in any of N slots within a set, and a fully associative cache allows any slot at all. Higher associativity reduces conflict misses at the cost of more complex (and slower) lookup hardware; most L1 caches settle on 4-way to 8-way associativity as the practical sweet spot.
Once there is more than one cache (one per core, typically), copies of the same line can exist in multiple places at once, and a write in one core's cache has to be made visible to the others. This is the job of Cache coherence, implemented either by snooping a shared bus (Cache snoopy protocols) or by tracking sharers explicitly (Cache directory protocols).