Nice insights, thanks! 1) Great point! This is def true for InstructGPT (they even say that explicitly in the paper) but is it also true for ChatGPT? A chat consists of multiple Q/A, where some As can get a +1 reward and others a -1, thus it seem