That's the danger with the transformer models. I also came across occasions where the result was wrong but ChatGPT sounded confident like an authority to trust. Together with the built-in bias it's quite concerning. should at least show confidence value for each answer.