The thing is you cant stump a large language model like ChatGPT unless you have an external oracle that can check its answers. You can ask it nonsensical things and it will answer you, complete with fake citations if you ask for them.