Agreed. Seq2seq does a nice job imitating reasoning (I don't think it truly reasons). With ChatGPT, some arithmetic output are obvious hallucinated token sequences (crazy numbers like 3.14.159). Others could be the result of expression evaluation. Dunno.