Fact that a language models performance is a function of the models size was shown a while ago. We also had some workarounds for 4096 context widths via careful prompt engineering. Still exciting to see the hype around ChatGPT though