model routing
Model routing is the practice of sending each request to one of several AI models based on factors such as task difficulty, required quality, cost and latency.
Model routing sends each incoming request to one of several AI models for processing. The choice depends on criteria such as task difficulty, required quality, cost and response time, and the component that makes the decision is called a router.
Services built on large language models use it to balance cost and quality, handing simple requests to smaller, cheaper models and harder ones to more capable models. It differs from the routing inside a mixture-of-experts model, which directs inputs among expert subnetworks within a single model rather than choosing among separate models.
This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.
Articles covering this entry
For teams, consider automatic model routing and a pooled token budget rather than fixed per-person quotas.
The harness also needs model routing: choosing an AI model for a task based on the performance required and the price of using it.
That separation is the point of model routing: choosing which model should handle a request before sending it along.