Direct Preference Optimization (DPO) is a method for aligning language models with human preference data, introduced by Stanford University researchers in 2023. Given pairs of preferred and rejected responses to the same prompt, it trains the model directly with a loss that raises the likelihood of the preferred response, without a separate reward model or reinforcement learning.
It is a simplified alternative to RLHF and, being easy to implement and stable to train, is widely used in the post-training of open-source LLMs.