RLHF (Reinforcement Learning from Human Preferences) is the main innovation made ChatGPT appear "human" compared to the previous GPT-3 model. released some RLHF training data on github: WebGPT Comparisons