Similar example, i explained how to read a Nethack observation (an ascii map of the dungeon) to ChatGPT. I asked it to navigate to the door by selecting a valid action (up, down, etc). When it chose an action, i gave it the next observation. It