Inference is the process of running a trained AI model to produce output for new input. A large language model generating a response to a prompt or an image model classifying a photo are both examples of inference. It is distinguished from training, in which a model's parameters are adjusted using data.
Whereas training is a large one-time use of compute, inference is repeated for every request while a service is running, so its cost grows with the number of users. Reducing the latency and cost of inference is therefore a central concern in operating AI services, addressed through techniques such as quantization, distillation and dedicated accelerator chips. In logic and statistics, the word refers to the process of drawing conclusions from evidence.