speculative decoding

An inference technique that speeds up LLM generation by having a small draft model propose tokens that the large model verifies in parallel.

3 articlesLast mentioned

Speculative decoding is an inference technique that speeds up text generation by large language models. A small, fast draft model first proposes several next tokens, and the large target model verifies the candidates in parallel in a single forward pass, accepting the matching prefix and replacing the first mismatch with its own prediction. With a suitably designed acceptance rule, the output distribution is identical to that of the target model generating alone, so quality is preserved while the number of sequential steps falls.

The method became widely known through papers from Google and DeepMind researchers published between 2022 and 2023. Autoregressive generation must run the whole model for each token and is therefore bound by memory bandwidth rather than compute; speculative decoding eases this bottleneck by committing several tokens per forward pass. Variants replace the separate draft model with multi-token prediction (MTP) heads attached to the target model or with n-gram guesses, and major inference engines such as vLLM and llama.cpp support the technique.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • DeepSeek Opens DeepInfer Serving Engine Alongside MIT-Licensed V4.1-FlashAI News ·

    DeepInfer also includes DSpark speculative decoding, asynchronous batch scheduling, and structured outputs that conform to JSON schemas, regular expressions, or…

    Read article
  • Local Qwen 3.8 27B Coding Test: Four Harnesses Fix Nearly the Same BugsAI Performance ·

    …uantized key and value caches, batch sizes of 4096 and 512, draft MTP speculative decoding with a maximum of two draft tokens, and the flag for chat templating.

    Read article
  • 2-bit Qwen 27B on a 16GB GPU: Strong at Apps and Games, Weaker in 3DAI Performance ·

    …ptional 2.93GB MTP (multi-token prediction) draft model can speed up generation through speculative decoding, where a helper model guesses several tokens ahead.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.