Models like ChatGPT are trained to maximize human approval. But maximizing approval could lead to models that just appear to be aligned, especially as they get better at fooling us. Pleasure to be part of this paper led by Richard: