ChatGPT hugely rely on unsupervised pretraining on hundreds of billions of tokens + supervised finetuning on a large amount of labeler written demos and labelled model generations. This seems very consistent with Yann's statement.