Obviously current models still make lots of errors when doing this. ChatGPT is especially interesting, since it was clearly trained to model a specific agent rather than the entire distribution, but can be tricked into modeling other agents with carefully designed contexts!