calibration

A machine learning concept describing how well a model's output probabilities match the actual frequency of outcomes.

14 articlesLast mentioned

Calibration describes how closely the probabilities a model outputs match the actual frequency of outcomes. If, among the cases a model assigns a probability of 0.8, roughly 80% turn out to be true, the model is well calibrated. Unlike accuracy, which only counts right and wrong answers, calibration concerns the reliability of the probability values themselves.

It matters in systems that compare a classifier's probability output against thresholds to decide between automatic handling and human review. Because calibration differs from model to model, a threshold chosen for one model cannot simply be carried over to another, and it is typically measured on evaluation data with known correct answers. In metrology, the term refers to adjusting a measuring instrument against a reference standard.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • GPT-6.2 Astra: Cheaper per Task and Strong in 3D, but Not the Top ModelAI News ·

    …I scored Astra at 29.01% on its RSI Index, an automated research benchmark calibrated so that 50% indicates human-level performance, compared with 27.07% for GP…

    Read article
  • NVIDIA CEO Exit, Claude Fable 5.2 and Adobe MAX: A Week That Reshaped AIAI News ·

    …emini Decisions API, built on a dedicated Gemini Decision 1 model, handles calibrated multimodal evaluations across text, images, video and PDF documents, scori…

    Read article
  • Interaction-as-a-Judge: Computer-Use Agents Test AI-Built Games and Web AppsAI Performance ·

    …of win rate over the base model, compared with 6.14 for the code-trained policy, which Arena's calibration equates to roughly a 60 to 70 point leaderboard jump.

    Read article
  • Decision Models: Probabilities, Not Text, for Faster and Cheaper Agent RoutingAI in Practice ·

    Given a piece of context and a fixed list of allowed answers, a decision model returns a calibrated probability for each answer in a single forward pass.

    Read article
  • RLHF-Heavy Models Write Stiffer Prose; Replay Production Traces Before SwappingAI Performance ·

    The CTO's view is that frontier labs have pushed this stage hard, optimizing for cautious refusals, and that the side effect is a rigidly calibrated voice.

    Read article
  • Prompt Audits and Cache Reuse Cut Claude Costs Up to 72.9%, Anthropic SaysAI in Practice ·

    This guidance is acknowledged to be somewhat overly prescriptive; well-calibrated multi-point scales can work if variance is tightly controlled.

    Read article
  • Claude Code Rebuilds a Multimeter's Firmware in Plain C From One PromptAI in Practice ·

    The EEPROM copy matters most: it holds each meter's factory calibration constants, which cannot be recreated if they are wiped.

    Read article
  • T-Shirt Brand Hands Support, Email, Design and Ads to Four AI AgentsAI Startups ·

    Once calibrated, the agent runs in a closed loop: it tracks campaign analytics, consults an e-commerce promotional calendar, and posts suggested campaigns to a…

    Read article
  • Claude Moves Into Google Docs as ChatGPT Starts Working Inside ChromeAI News ·

    Score A calibrated rating that places an input on a ladder of levels you define

    Read article
  • Krea 3 and Grok Voice 3.0 Stand Out in Hands-On Tests of New AI ModelsAI News ·

    Unlike generative models that write paragraphs, decision models evaluate an input and return a choice or a calibrated probability.

    Read article
  • Protein Structure AI's Next Frontier: Learning How Molecules Actually MoveAI Health ·

    Because AlphaFold 2's confidence was well calibrated, scientists could trust it, and the community later discovered an unplanned use: identifying intrinsically…

    Read article
  • Why Claude Code Gets More From the Same Model: Five Parts of a HarnessAI in Practice ·

    It is a promising idea, but it appears to need careful calibration, and most permission checks today are still procedural.

    Read article
  • AI Agents and Commerce: Stripe's John Collison on Discovery, APIs and SecurityAI News ·

    To stay calibrated, he prefers vibe coding, building small apps by describing them to an AI, and talking with fast-growing customers over reading stacks of rese…

    Read article
  • Navier-Stokes AI Claim: The Proof, the Limits, the ReviewAI News ·

    The pulses are calibrated so problematic nonlinear terms in the calculated force cancel.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.