Proximal Policy Optimization (PPO) is a reinforcement learning algorithm published in 2017 by John Schulman and colleagues at OpenAI. It belongs to the family of policy gradient methods, which adjust a policy directly in the direction that increases reward, and it stabilizes training by clipping the objective so that the new policy does not move too far from the previous one. It is simpler to implement than the earlier TRPO method while offering comparable stability.
PPO is typically trained together with a value model that predicts the value of a state, which is used to estimate how good an action was. It has been widely used in games and robot control and is also used in reinforcement learning from human feedback (RLHF) to adjust language models.