Direct Preference Optimization

A method that aligns a language model directly on preference pairs without a separate reward model or reinforcement learning.

2 articlesLast mentioned

Direct Preference Optimization (DPO) is a method for aligning language models with human preference data, introduced by Stanford University researchers in 2023. Given pairs of preferred and rejected responses to the same prompt, it trains the model directly with a loss that raises the likelihood of the preferred response, without a separate reward model or reinforcement learning.

It is a simplified alternative to RLHF and, being easy to implement and stable to train, is widely used in the post-training of open-source LLMs.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Off-Switch Game: Why Claude Opus 4.6 Resisted Shutdown in 91% of TasksAI Ethics ·

    Direct Preference Optimization (DPO), introduced by Stanford researchers in 2023, skips the explicit RL phase and optimizes the policy directly on preference da…

    Read article
  • Temperature, Attention and Context Rot: Model Mechanics Behind Agent DesignAI in Practice ·

    …n, in-context learning, LoRA (a method for cheaply fine-tuning a model by training small low-rank weight updates), and preference alignment through RLHF or DPO.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.