ChatGPT modifies GPT3 in two ways: 1) fine-tune generative model on human-written 2) * generate machine responses * humans rate 1-7 * train new neural net to approximate human ratings * fine-tune generative model to maximize those synthetic rating "rewards" That's it.