It's said that ChatGPT can give you the right answer, but it can't explain why it's right, or give a source. Exactly the same is true of humans, though. If you ask me how many neurons in a human brain, I'll say 88 billion, but I don't have the reference for that in my head.