I believe GPT-3.5 and ChatGPT have a context of 4096, which would explain how it's so coherent in comparison... I think I could theoretically fit a bigger context in the remaining memory on a local rig like this one.