It is happening already. ChatGPT was trained on Common Crawl 60% 410 billion tokens WebText2 22% 19 billion tokens Books1 8% 12 billion tokens Books2 8% 55 billion tokens Wikipedia 3% 3 billion tokens Most of the data is garbage c