For pre-training data, I think they get it all online. But, they do have a massive amount of data collected from interactions with GPT. See for example Sec 3.2 of the paper below. They (legally) use data from GPT APIs to train models like ChatGPT.