SGLang

SGLang is an open-source serving framework for large language models released by LMSYS researchers in 2024, known for RadixAttention-based prefix caching.

2 articlesLast mentioned

SGLang is an open-source framework for fast serving of large language models and vision-language models. It was released in 2024 by LMSYS researchers centered at UC Berkeley and is distributed under the Apache 2.0 license.

Its key features include RadixAttention, which manages the KV cache of shared prompt prefixes in a radix tree and reuses it automatically across requests; fast generation of structured output constrained by regular expressions or JSON schemas; and a frontend language for expressing model calls and control flow in Python. Alongside vLLM, it is one of the most widely used LLM inference engines, and it also supports disaggregated prefill and decode serving and expert parallelism for large MoE models.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Temperature, Attention and Context Rot: Model Mechanics Behind Agent DesignAI in Practice ·

    SGLang compiles the schema into a finite state machine, inserts fixed syntax such as quotes and braces without a model forward pass, and only invokes the model…

    Read article
  • DeepSeek Opens DeepInfer Serving Engine Alongside MIT-Licensed V4.1-FlashAI News ·

    On October 26, Cloudflare published a post-mortem describing an SGLang bug in disaggregated serving that leaked 155 responses containing other customers' data o…

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.