Plug in a model's dimensions and a GPU, and this analyzer computes the arithmetic intensity of the prefill and decode phases, places each on the hardware roofline, and solves for the exact batch size where decode stops wasting your tensor cores.
| Metric | Value |
|---|---|
| GPU ridge point (I*) | — |
| Decode arithmetic intensity | — |
| Achievable decode TFLOP/s (roofline cap) | — |
| Min batch to make decode compute-bound | — |
| Est. decode weight-read time / step | — |
The roofline model answers one question: for a given operation, is the GPU limited by how fast it can read data from memory, or by how fast it can do math? The dividing line is arithmetic intensity — the ratio of floating-point operations to bytes moved from HBM:
AI = FLOPs / bytes_moved
Each accelerator has a ridge point, I* = peak_FLOPS / memory_bandwidth (both in matching units). If a kernel's intensity is below I* it is memory-bandwidth-bound: the tensor cores sit idle waiting on HBM, and the best achievable throughput is AI × bandwidth. Above I* it is compute-bound and runs at peak FLOP/s.
For a transformer layer the dominant matmuls are the QKV, attention output, and two MLP projections. A useful approximation for the linear layers is FLOPs ≈ 2 × batch × tokens × params_per_token and bytes ≈ dtype × (weights + activations). During prefill all prompt tokens stream through the weights at once, so every weight byte is reused across batch × seq tokens — intensity is high and prefill is almost always compute-bound. During decode you generate one token per request per step, so each weight matrix is read from HBM to serve only batch tokens; intensity scales roughly linearly with batch, which is why single-request decode is severely memory-bound and why batching is the primary throughput lever.
The analyzer solves AI(batch) = I* for the crossover batch size — the point where decode finally saturates compute. Below it, adding batch is nearly free throughput; above it you are compute-limited and latency starts to climb. The numbers are first-order estimates (they ignore attention-score FLOPs beyond the KV read, kernel-fusion, and cache effects) but they capture the shape that governs real deployment decisions.
KV-Cache VRAM Calculator LLM Throughput Estimator GPU VRAM Fit Checker