TensorRT-LLM is an open-source library released by NVIDIA in 2023 for accelerating large language model (LLM) inference on NVIDIA GPUs. It is built on top of TensorRT, NVIDIA's deep learning inference optimizer, and lets developers define models through a Python API that is then compiled into execution engines optimized for the target GPU.
Key features include in-flight batching, which adds new requests to a running batch as they arrive, efficient management of the KV cache used in attention computation, quantization formats such as FP8 and INT4, and tensor parallelism for splitting a model across multiple GPUs. It supports widely used open models including the Llama family, GPT-style models and Mistral, and is often deployed together with NVIDIA Triton Inference Server. In enterprise settings running data-center GPUs, it is regarded alongside vLLM as one of the main LLM inference engines.