I wonder if this is an artifact of training to work as a chatbot. Other GPT variants, including the RLHF ones like text-davinci-003 don't answer this this specific question like this ChatGPT seems to have learned to "guess the teacher's password"