Unlike GPT-3 or PaLM, ChatGPT is not just a raw language model trained with maximum likelihood. It's fine-tuned with a combination of additional supervised and reinforcement learning signals based on human preferences. Video covering the training recipe: