We don't know the context window length... But even chatgpt is speculated to be longer than 4k. Maybe it does forget. Maybe it does summaries for previous conversations. See for a few, already existing techniques on this.