In ChatGPT, the authors developed a model that takes in a sequence of text and returns a scalar reward which represent human preference. The reward model is trained with data obtained from human annotators and aims to capture human judgment.