A utility function assigns a real number to each possible outcome or state, expressing how desirable it is. The concept comes from economics and decision theory, where a rational agent is assumed to choose actions that maximize expected utility.
In AI, utility functions are used to specify an agent's objective mathematically. Unlike a reward function in reinforcement learning, which provides a reward at each step, a utility function expresses the value of an overall outcome. AI safety research examines problems that arise when a designed utility function diverges from human intent, as well as designs in which the agent remains uncertain about the human's utility.