reward misspecification

Reward misspecification is a problem in reinforcement learning and AI alignment in which a reward function or evaluation measure does not accurately represent the designer’s goal. A model can score highly on the specified measure while producing an unintended outcome. It concerns how the measure is defined, whereas reward hacking describes behavior that exploits its weaknesses.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.