Things like ChatGPT will just return whatever has the highest probability score accroding to the model, and it would be very naive to take it's word for it. It's not even very good as information retrieval tool since it can easily mess up things it hasn't seen much of.