Main system RAM is fast in absolute terms, but glacially slow compared to a modern CPU core’s actual clock speed. CPU cache exists to bridge that gap — a series of progressively larger, progressively slower memory tiers sitting directly between the core and RAM.
The Three-Tier Model
- L1 — smallest (typically 32-64KB per core) and fastest, split between instructions and data, accessible in just a few CPU cycles.
- L2 — larger (256KB-2MB per core), still per-core on most designs, somewhat slower than L1.
- L3 — largest (8-96MB+), typically shared across all cores on the chip, meaningfully slower than L1/L2 but still dramatically faster than main RAM.
Approximate access latency, in CPU cycles: L1 cache: ~4 cycles L2 cache: ~12 cycles L3 cache: ~40 cycles Main RAM: ~200+ cycles A cache miss that goes all the way to RAM can cost 50x the latency of an L1 hit — this is why cache-friendly code matters so much for performance-critical work.
Why Cache-Friendly Code Is Real, Not Folklore
#define SIZE 1000
int matrix[SIZE][SIZE];
// Cache-unfriendly: jumps across memory, poor locality
void sum_columns_first() {
long sum = 0;
for (int col = 0; col < SIZE; col++)
for (int row = 0; row < SIZE; row++)
sum += matrix[row][col]; // jumps SIZE elements each iteration
}
// Cache-friendly: accesses memory sequentially, matching how it's laid out
void sum_rows_first() {
long sum = 0;
for (int row = 0; row < SIZE; row++)
for (int col = 0; col < SIZE; col++)
sum += matrix[row][col]; // sequential access, cache-friendly
}
These two functions do mathematically identical work but can differ by several times in actual runtime, purely because of memory access pattern — the second version keeps pulling data the CPU already has cached nearby, while the first constantly triggers slow cache misses.
Why This Still Matters in High-Level Languages
Even in Python or JavaScript, the underlying memory layout of your data structures affects performance through exactly this mechanism — it’s why NumPy arrays (contiguous, cache-friendly memory) dramatically outperform equivalent operations on plain Python lists of objects scattered throughout memory.