Worth noting that this stops being the case with any models which are tuned on human preferences (InstructGPT, now the default GPT model of OpenAIs API, ChatGPT, etc). You are now asking something more akin to, what would a human prefer as