RLHF / RLAIF seems to be the answer. Instruction finetuning gave more attention to the instructions but the generated texts often fall short of our expectations. Comparing text-davinci-003 to ChatGPT shows that RL tries to solve the problem.