I don't think that's the case anymore. This was still valid with GPT3 when it was released. You could clearly see how the engineers were tweaking the model to make it solve word problems and instructions. ChatGPT which should be a GPT3.5 model