IMO what makes ChatGPT so effective it's the instruct fine-tuning + RLHF done at scale. The NNs haven't changed much since Transformers - essentially Sutton's bitter lesson. If you extrapolate that, the near term problem is lack of open