Even after all that fancy RLHF work into aligning with "values", this seems like an unfortunate side-effect. Maybe with all the data ChatGPT is collecting right now, it can RL its way out of the RL hole it trained right into.