Compute-Bound and Memory-Bound
Every operation needs two things: it needs to move data (load weights, read inputs, write outputs) and it needs to do arithmetic (multiply, add). Hardware has a maximum rate for each: memory bandwidth (bytes/second) and compute throughput (FLOPs/second).
If the operation finishes its arithmetic before the data has arrived, it is memory-bound — the compute units sit idle waiting for memory. If data arrives faster than the hardware can process it, the operation is compute-bound — the memory system sits idle waiting for the ALUs.
The dividing line is the hardware's operational intensity threshold: the ratio of peak FLOPs/second to peak bytes/second. If your operation's arithmetic intensity (FLOPs per byte moved) is below this threshold, it is memory-bound. If above, it is compute-bound.
A key insight: an operation can have very few total FLOPs and still be slow if it is memory-bound. A single decode step through a 7B-parameter model does relatively little arithmetic per weight, but must load gigabytes of weights through the memory bus.
The roofline model is a visual tool for understanding performance limits. It plots achievable performance (FLOP/s) against arithmetic intensity (FLOP/byte). The plot has two regimes:
On the left slope, performance is limited by memory bandwidth: the hardware can compute faster than data arrives. Every byte you save (through quantization, smaller types, better layout) directly improves speed. On the right flat region, performance is limited by compute: data arrives faster than the hardware can process it. Better algorithms, SIMD utilization, and kernel optimization help here.
The machine balance point (also called the "ridge point") is where the slope meets the ceiling. Its x-coordinate is peak_FLOP/s ÷ peak_bandwidth — the hardware's characteristic ratio. Operations to the left of this point are memory-bound; operations to the right are compute-bound.
The same weight matrix, the same kernel — but with different batch sizes, the operation can be either memory-bound or compute-bound. This is the single most important performance concept in inference:
This is why prefill (hundreds of tokens) is compute-bound and decode (one token) is memory-bound, even though they execute the same operations on the same weights. The batch size determines which resource is the bottleneck.
It also explains why batching multiple users' decode steps together (continuous batching) improves throughput: combining 32 single-token decodes into one batch of 32 shifts the arithmetic intensity from ~1 to ~32 FLOP/byte, potentially moving the operation from deep in the memory-bound region toward the ridge point.
Machine balance varies enormously across hardware. This determines which operations are memory-bound on your specific system:
On the A100, an operation needs 153+ FLOP/byte to be compute-bound — single-token decode at ~1 FLOP/byte is 150× below the ridge point. On a CPU with lower balance (~5), the same operation is still memory-bound but less extremely so.
Hardware: 10 TFLOP/s compute, 100 GB/s memory bandwidth.
Threshold (machine balance) = Peak FLOP/s ÷ Peak bytes/s = 10T ÷ 100G = 100 FLOP/byte.
If an operation does more than 100 FLOPs per byte loaded, compute is the bottleneck. If fewer, memory is.
Operation B does 60× fewer FLOPs than A but takes longer, because it is memory-bound.
In llama.cpp, whether a ggml_mul_mat call ends up compute-bound or memory-bound depends on the input batch size. The same kernel, the same weights — but with n_tokens = 1 (decode) the operation is memory-bound, and with n_tokens = 512 (prefill) it is compute-bound. Backend code may choose different execution strategies based on these shapes.
Correctly classifying your bottleneck changes your optimization strategy entirely. For compute-bound work, you want better SIMD utilization, tiling, and instruction scheduling. For memory-bound work, you want to reduce bytes moved: quantization, smaller data types, and better memory access patterns. Applying the wrong optimization category wastes effort.
An operation does 50 million FLOPs and moves 200 MB of data. On hardware with a machine balance of 100 FLOP/byte, is it compute-bound or memory-bound?
You quantize weights from FP16 to INT8, halving the bytes moved. For a memory-bound decode step, what speedup do you expect? For a compute-bound prefill step?