Speculative Decoding

Fri, 18 Sep 2026 13:56:17 +0000

Speculative decoding is a technique in AI inference optimization where we combine a fast draft model usually smaller in parameter size with the actual large language model to help speed up token generation time. It’s a very useful and widely adopted technique in inference engineering and optimization. Let’s look at how this works and tradeoff one needs to make in order to use speculative decoding in their inference pipeline.

How Speculative Decoding Works

Speculative decoding introduces a small parameter model that can work and generate next token faster than the large LLM model with reasoning capability. The small model is termed as “Draft Model” in the diagram below. The draft model generates sequence of tokens and passes it to the actual model to evalulate and verify. LLM evaluates the tokens and accepts the token which is correct and either replaces or rejects the tokens which are wrong. All of this happens in one forward pass which saves times and improves performance. Post verification stage LLM updates the KV cache with the accepted tokens and proceeds to generate more tokens required to process the prompt. This approach increases TPS (token per second) aggresively.

Ai on Shahid Shaikh

Speculative Decoding

How Speculative Decoding Works