models like chatGPT are doing many layers of mesa-optimization (hierarchical optimizers). the actual value function is this soft mushy thing distributed over 1000s of human reviewers, approximated by a supervised reward model, approximated internally by GPT-3 to do well on PPO