In the end, using these reward models, the OpenAI team fine-tuned the model using Proximal Policy Optimization (PPO). Keep in mind that ChatGPT is fine-tuned from a model in the GPT-3.5 series, which finished training in early 2022.