reward model

A reward model is a model trained to score how good an AI model's output is, used to produce the reward signal in reinforcement learning methods such as RLHF.

3 articlesLast mentioned

A reward model is a model trained to assign a numerical score indicating how desirable an AI model's output is. When a language model is tuned with reinforcement learning, humans cannot evaluate every output, so a reward model that has learned human preferences produces the reward signal in their place.

The best-known use is in reinforcement learning from human feedback (RLHF). Human raters rank or compare multiple outputs for the same prompt, the reward model is trained on this preference data, and the language model is then optimized to maximize the reward model's score. Reward models are sometimes divided into outcome reward models, which judge only the final result, and process reward models, which judge each step of reasoning. If a reward model captures human intent imperfectly, the policy may exploit its flaws, a behavior known as reward hacking.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Off-Switch Game: Why Claude Opus 4.6 Resisted Shutdown in 91% of TasksAI Ethics ·

    The preferred and rejected outputs train a reward model using the Bradley-Terry formulation, where the probability that one output beats another is e^(rp) / (e^…

    Read article
  • Interaction-as-a-Judge: Computer-Use Agents Test AI-Built Games and Web AppsAI Performance ·

    The second is a trained reward model built on Qwen-3.5-9B, with a vision encoder for the frames, text embeddings for code, actions and reasoning, and a linear h…

    Read article
  • ChatGPT for Financial Services: One Shared Model, Hosted Data, Human ReviewAI News ·

    …g from human feedback (RLHF), in which human labelers rank several model outputs from best to worst and a reward model is trained on those pairwise comparisons.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.