holdout

A data set kept apart from training and tuning so that it can be used only for final evaluation.

2 articlesLast mentioned

In machine learning, holdout refers to an evaluation method in which a portion of the available data is set aside and not used for training or tuning, and also to the data set reserved in this way. After training, performance is measured on the unseen holdout set to estimate how well the model generalizes to new data and to detect overfitting to the training data.

It is the simplest form of data splitting, dividing data once into training, validation, and test portions, in contrast to cross-validation, which rotates through multiple splits. When evaluating AI systems or agents, undisclosed holdout task sets are likewise used to prevent repeated tuning against the same benchmark.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Prompt Audits and Cache Reuse Cut Claude Costs Up to 72.9%, Anthropic SaysAI in Practice ·

    â–² Hillclimbing on a train split with a held-out test set

    Read article
  • Self-Improving Agent Design: Two Loops, Versioned Harnesses, Sealed EvaluatorsAI in Practice ·

    Refresh the holdout set continuously so that it tracks real production workloads and the agent cannot adapt to a fixed evaluation footprint.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.