Per Open Ai's website, if you build with InstructGPT and dont instruct the language model to be polite, ideally with examples, "things totally go off the rails"...Whomever instructs the model sets the bias.