This only works if the training data aligns with reality. But even playing with ChatGPT now you can tell it knows forbidden knowledge it is not allowed to talk about. Programmers have very obvious delusions