Calibration describes how closely the probabilities a model outputs match the actual frequency of outcomes. If, among the cases a model assigns a probability of 0.8, roughly 80% turn out to be true, the model is well calibrated. Unlike accuracy, which only counts right and wrong answers, calibration concerns the reliability of the probability values themselves.
It matters in systems that compare a classifier's probability output against thresholds to decide between automatic handling and human review. Because calibration differs from model to model, a threshold chosen for one model cannot simply be carried over to another, and it is typically measured on evaluation data with known correct answers. In metrology, the term refers to adjusting a measuring instrument against a reference standard.