If InstructGPT numbers could be considered a lower bound for the training of ChatGPT, it suggests the labeling effort for RL training is indeed like "the cherry on the cake" except the cake is kind of useless without the cherry for dialog interaction with a LLM