Reinforcement learning from human feedback (RLHF) is the main workhorse behind models like InstructGPT, ChatGPT and Sparrow. A key assumption that underlies all this work is that human preferences are static and homogeneous. (2/n)