for as long as possible (quite the contrary, actually). Regarding structure, I see your point - since GPT-3.5 (or whatever was the base model for ChatGPT) also learns to output bad (unhelpful or harmful) answers, RLHF has the goal to push it away