I think it's the latter. The key challenges with foundational models (such as stochastic parroting) have not been solved with more data. See journey from GPT1 to ChatGPT. Bengio Kiela are kicking off the new year by focusing on smarter mode