Large language models like ChatGPT can't actually perform math; they basically have to "guess" the answer. Code also has the benefit of having well defined patterns and plenty of examples in the training data.