See the paragraph "We trained this model using Reinforcement Learning from Human Feedback (RLHF)" in the original ChatGPT launch post: The InstructGPT post that links to is pretty important too - that's the core answer