The memory window is a good idea, but ChatGPT is doing something more advanced. For the prompt "Let's play a game. I'll give you some numbers, and you'll repeat them back to me.", ChatGPT will happily remember everything I've given it (whereas this is bounded by the window size).