ExLlamaV2 is an open-source inference library designed to run quantized large language models (LLMs) efficiently on consumer-grade NVIDIA GPUs. It was released in 2023 by a developer known on GitHub as turboderp as the successor to the original ExLlama project. The library is written in Python, C++ and CUDA and supports a range of open-weight models, including the Llama family.
ExLlamaV2 introduced its own quantization format, EXL2, which allows different layers and weights within a model to use different bit widths. This makes it possible to fine-tune the average bits per weight and fit large models into limited GPU memory. Alongside tools such as vLLM and llama.cpp, it is one of the inference engines commonly used by hobbyists and small-scale operators running LLMs locally.