The thing about ChatGPT is it's already an LLM that is in some sense simulating how a thinking entity responds. If that simulation of a thinking entity gets good enough, then I suspect (with caveats) that this becomes indistinguishable from