Case Study: Interpreting a Prefill vs Decode Profile

A profile shows you where the runtime actually goes — but reading one correctly is a skill. The most common mistake is jumping to a local conclusion from the single hottest function. Instead, you need to look at the distribution of time across operator types and ask what that distribution tells you about the system's bottleneck.

Prefill and decode have fundamentally different profiles:

• Prefill Processes many tokens at once. Large matrix multiplies dominate. Compute-bound. The hotspot is typically in GEMM (general matrix multiply) kernels.
• Decode Processes one token at a time. Each operation loads large weight matrices for a tiny computation. Memory-bandwidth-bound. The hotspot is weight loading, not arithmetic.

A profile that shows 80% of time in matrix multiply kernels during prefill is normal — that is the GPU doing useful compute. The same profile during decode might mean the kernel is not the bottleneck; the time is spent waiting for data to arrive from memory.

The key habit: look at the distribution first, hypothesize the bottleneck class (compute vs memory vs overhead), then drill into individual functions. Never start with the hottest function and assume that is "the problem."

A profile is a table of functions (or operations) and the time spent in each. Here is a systematic approach to interpreting one:

1. Classify the mode. Is this a prefill profile or a decode profile? Check the batch size or token count. If you do not know, the profile distribution itself tells you: if GEMM dominates (>60%), it is likely prefill. If GEMV + memory ops dominate, it is likely decode.
2. Sum the categories. Group operations into compute (GEMM/GEMV), memory (KV cache reads/writes, weight loads), normalization (RMSNorm, softmax), and overhead (kernel launch, sync, memory allocation). The category totals reveal the bottleneck class.
3. Check the overhead fraction. In a healthy profile, overhead should be <15% for prefill and <25% for decode. High overhead (>30%) suggests problems: too many small kernel launches, excessive synchronization, or memory allocation during inference.
4. Compare against expected counts. Derive the expected kernel count from your model's block structure: count the matmuls per layer (Q/K/V projections, output projection, FFN matrices) and multiply by the number of layers. If the profile shows significantly more calls than expected, investigate — something may be computed redundantly. If certain layers take much longer than others, check for heterogeneous config (MoE, different attention type).
5. Only then look at individual hotspots. Now that you understand the big picture, drill into the top 3-5 functions. Ask: is this function slow because of its inherent cost, or because it is being called more often than expected?

After reading many profiles, certain patterns become recognizable:

• Healthy prefill: 65-80% GEMM, 5-10% softmax, <15% overhead. The compute units are busy doing useful work. Optimization targets: GEMM kernel efficiency, tiling strategy.
• Healthy decode: 50-65% GEMV, 15-20% KV cache, <25% overhead. Memory bandwidth is saturated. Optimization targets: quantization (reduce bytes loaded), KV cache layout.
• Overhead-dominated: >30% in sync/launch/alloc. This is pathological — the hardware is mostly idle. Causes: too-small batch sizes, un-fused operations creating many tiny kernels, memory allocation during the hot loop instead of pre-allocation.
• Softmax-heavy prefill: >15% in softmax. This usually means very long sequences — the softmax cost scales with n_tokens² while projection GEMMs are linear in n_tokens (for fixed model width). At extreme sequence lengths, attention score computation and softmax can rival the projection GEMMs.

A profile tells you where the time goes. It does not directly tell you what to do. The translation depends on the bottleneck class:

Applying the wrong category of optimization wastes effort. Optimizing SIMD instructions when you are memory-bound will have negligible effect — the ALUs are already waiting for data.

Arithmetic intensity determines the bottleneck regime: