A reward model is a model trained to assign a numerical score indicating how desirable an AI model's output is. When a language model is tuned with reinforcement learning, humans cannot evaluate every output, so a reward model that has learned human preferences produces the reward signal in their place.
The best-known use is in reinforcement learning from human feedback (RLHF). Human raters rank or compare multiple outputs for the same prompt, the reward model is trained on this preference data, and the language model is then optimized to maximize the reward model's score. Reward models are sometimes divided into outcome reward models, which judge only the final result, and process reward models, which judge each step of reasoning. If a reward model captures human intent imperfectly, the policy may exploit its flaws, a behavior known as reward hacking.