I think the Chinese Room argument leads in this direction, too. ChatGPT doesn't /really/ understand truth and reality. You can see this in the way that it can produce incorrect output yet argue confidently and persuasively for its error.