vLLM

An open-source library for LLM inference and serving released in 2023 by researchers at UC Berkeley, known for its PagedAttention technique.

4 articlesLast mentioned

vLLM is an open-source library for large language model inference and serving. Released in 2023 by researchers at UC Berkeley, it introduced PagedAttention, a technique that manages the attention KV cache in blocks much like virtual memory pages in an operating system, reducing wasted GPU memory and substantially increasing throughput.

It supports a wide range of models published on Hugging Face and provides an OpenAI-compatible API server so that existing applications can use it directly. It also supports continuous batching, quantization, and distributed inference across multiple GPUs, and is one of the most widely used inference engines for self-hosting open-source LLMs.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Temperature, Attention and Context Rot: Model Mechanics Behind Agent DesignAI in Practice ·

    …ss, and only invokes the model at transitions where several tokens are valid; this yields substantial throughput gains over vLLM, Guidance and LMQL on Llama-7B.

    Read article
  • Claude Fable 5.2 Benchmarks and Five Real Tasks: Where the Gains Show UpAI Performance ·

    Fable 5.2 was asked to survey hardware prices in Europe and Portugal, memory needs, and inference engines including vLLM, TensorRT-LLM and ExLlamaV2.

    Read article
  • Coder's Hybrid AI Gateway: Local Models for Bots, Cloud for Developer ChatAI Performance ·

    An engineer installs Ollama or vLLM on a local machine.

    Read article
  • Intelligence Plane: Why Memory and Ontology, Not the Model, Define AI AgentsAI in Practice ·

    …ervices Container runs on premises or air-gapped, packaging LLM inference via vLLM or llama.cpp on CPUs or GPUs, embeddings, reranking and text classification b…

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.