pretraining

Pretraining is the stage in which a model first learns general capabilities from large-scale data. For large language models, the typical approach is learning to predict the next token across vast amounts of text. The resulting model is then adapted to specific uses through fine-tuning or further training.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.