Speculative decoding is an inference technique that speeds up text generation by large language models. A small, fast draft model first proposes several next tokens, and the large target model verifies the candidates in parallel in a single forward pass, accepting the matching prefix and replacing the first mismatch with its own prediction. With a suitably designed acceptance rule, the output distribution is identical to that of the target model generating alone, so quality is preserved while the number of sequential steps falls.
The method became widely known through papers from Google and DeepMind researchers published between 2022 and 2023. Autoregressive generation must run the whole model for each token and is therefore bound by memory bandwidth rather than compute; speculative decoding eases this bottleneck by committing several tokens per forward pass. Variants replace the separate draft model with multi-token prediction (MTP) heads attached to the target model or with n-gram guesses, and major inference engines such as vLLM and llama.cpp support the technique.