How much faster is speculative decoding for your setup?
Speculative decoding uses a small draft model to propose several tokens that a large target model verifies in a single forward pass. This calculator predicts the real wall-clock speedup from your draft acceptance rate, the speculation length, and how much cheaper the draft model is per token.
| γ | E tokens | Speedup |
|---|
The formula behind the number
Each verification step, the draft model proposes γ tokens. Under the standard model each token is independently accepted with probability α, so acceptance stops at the first rejection. The expected number of tokens produced per step — accepted drafts plus the one bonus token the target always emits — is a truncated geometric series:
The cost per step, measured in target-model forward passes, is one full target verification pass (cost 1) plus the draft model generating γ tokens at a per-token cost c relative to the target. That gives a wall-clock speedup versus vanilla decoding of:
Two forces pull against each other. Raising γ lets a single verification pass accept a longer run of tokens, but every extra drafted token costs c whether it is accepted or not, and acceptance decays geometrically. The tool sweeps γ from 1 to 12 and reports the optimal speculation length for your α and c — the point where marginal draft cost overtakes the marginal accepted tokens. High acceptance and a cheap draft (small c) favor longer speculation; a mediocre draft or an expensive one favors γ = 1–2. If the speedup drops below 1.0, speculative decoding is a net loss and you should decode normally. Multiply the speedup by your measured baseline tokens/sec to get an effective throughput estimate; treat it as an upper bound, since batching, KV-cache pressure, and drafting overhead in real serving stacks shave off some of the theoretical win.