ChatGPT got the answer right when the query was a single token, and wrong when the query crossed a token boundary. Not sure whether this is reproducible or just luck (or even whether the GPT-3 tokenizer is also used by ChatGPT), but it seems like it would make sense.