Models like chatGPT are trained through two steps: 1. Train on unlabelled text data from all over the internet 2. Finetune using preferences from human annotators The human annotators follow "annotation guidelines" but their judgement effectively determines the model's "values"