An AI gateway is a server layer placed between applications and multiple AI model APIs that relays every request. It is also called an LLM gateway. It unifies several model providers behind a common API and centrally handles authentication and access control, usage metering, cost management, request logging and auditing, routing and fallback between models, rate limiting, and caching.
It extends the API gateway concept to LLM calls and is used by organizations to apply policies without modifying individual applications when switching providers or combining multiple models. Cloudflare AI Gateway, Portkey, LiteLLM, and Bifrost fall into this category.