It's tricky to evaluate chatGPT on coding problems. You need to make sure that on the problems you're testing it, they were **not** used for training. In the AlphaCode paper: they did a pretty good evaluation.