A nice article examining the RL approaches to make a LLM into a Conversation agent like ChatGPT. ChatGPT is most likely doing what InstructGPT is doing - outputting the entire sentence as action as opposed to evaluating reward for each token like in AR modeling of first stage