People believe that RLHF is magical (which it is to some extent), but they will be disappointed when they realize its limitations. ChatGPT is reliant on three types of data: Prompts, human responses to sampled prompts Human ratings to ma