TensorRT-LLM

TensorRT-LLM is an open-source library released by NVIDIA in 2023 that optimizes large language model inference on NVIDIA GPUs.

1 articleLast mentioned

TensorRT-LLM is an open-source library released by NVIDIA in 2023 for accelerating large language model (LLM) inference on NVIDIA GPUs. It is built on top of TensorRT, NVIDIA's deep learning inference optimizer, and lets developers define models through a Python API that is then compiled into execution engines optimized for the target GPU.

Key features include in-flight batching, which adds new requests to a running batch as they arrive, efficient management of the KV cache used in attention computation, quantization formats such as FP8 and INT4, and tensor parallelism for splitting a model across multiple GPUs. It supports widely used open models including the Llama family, GPT-style models and Mistral, and is often deployed together with NVIDIA Triton Inference Server. In enterprise settings running data-center GPUs, it is regarded alongside vLLM as one of the main LLM inference engines.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Claude Fable 5.2 Benchmarks and Five Real Tasks: Where the Gains Show UpAI Performance ·

    Fable 5.2 was asked to survey hardware prices in Europe and Portugal, memory needs, and inference engines including vLLM, TensorRT-LLM and ExLlamaV2.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.