Related: I talked in depth with InstructGPT ( "sibling model" to ChatGPT) co-author and PPO inventor John Schulman about InstructGPT, his takes on RLHF on LLMs, etc. I learned some things here I hadn't heard anywhere else.