A lot of the wow behind chatGPT is still with the underlying language model GPT-3. That is a transformer based model trained on a bunch of data from wiki, books and very important Common Crawl which is a repo of web scraped data