Context size for chatgpt is already 8K, will probably be rolled out to other models soon. Issue with training larger context models is finding quality large-context data. Definitely a strong moat, but only for certain use-cases, and there are work-