In machine learning, holdout refers to an evaluation method in which a portion of the available data is set aside and not used for training or tuning, and also to the data set reserved in this way. After training, performance is measured on the unseen holdout set to estimate how well the model generalizes to new data and to detect overfitting to the training data.
It is the simplest form of data splitting, dividing data once into training, validation, and test portions, in contrast to cross-validation, which rotates through multiple splits. When evaluating AI systems or agents, undisclosed holdout task sets are likewise used to prevent repeated tuning against the same benchmark.