ExLlamaV2

ExLlamaV2 is an open-source inference library for running quantized large language models efficiently on consumer NVIDIA GPUs.

1 articleLast mentioned

ExLlamaV2 is an open-source inference library designed to run quantized large language models (LLMs) efficiently on consumer-grade NVIDIA GPUs. It was released in 2023 by a developer known on GitHub as turboderp as the successor to the original ExLlama project. The library is written in Python, C++ and CUDA and supports a range of open-weight models, including the Llama family.

ExLlamaV2 introduced its own quantization format, EXL2, which allows different layers and weights within a model to use different bit widths. This makes it possible to fine-tune the average bits per weight and fit large models into limited GPU memory. Alongside tools such as vLLM and llama.cpp, it is one of the inference engines commonly used by hobbyists and small-scale operators running LLMs locally.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Claude Fable 5.2 Benchmarks and Five Real Tasks: Where the Gains Show UpAI Performance ·

    Fable 5.2 was asked to survey hardware prices in Europe and Portugal, memory needs, and inference engines including vLLM, TensorRT-LLM and ExLlamaV2.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.