ChatGPT v.0.1 was then trained to optimize this reward function using an algorithm called PPO. In a nutshell, it tries to generate responses that maximizes the reward predicted by the reward model above.