Surprisingly, the latest Open AI models: chatGPT and text-davinci-003 are actually finetuned from the code generation models, not pure text generation. There's a lot more detail on what base models are used in the model index for researchers: