You can't simply get a bunch of value-neutral data for pre-training but can control the smaller fine-tuning dataset with much more granularity, and we do that/are getting better over time (see InstructGPT paper, ChatGPT blog post). Also ne