tl;dr: mesa-optimization is one justification for claiming that Transformers' generic capabilities generalize more readily than their alignment with the specific reward function that they were trained on. (Direct experience interacting with Cha