Ive had the same experience. ChatGPT clearly needs some kind of RL post-training that it can run through stockfish as a compile step. The bright side is that GitHub copilot has shown this is totally doable as it rarely output