ChatGPT's RLHF stage loss function is likely to be similar to the one in InstructGPT paper. Image from blog post ChatGPT may have traded off NLP SOTA performance also in-context learning according to blog) so the last term may be zero