The ChatGPT people worked around this partially by essentially finetuning GPT-3 on lots of "judges", to the point of partial modal collapse (not modeling the input training distribution accurately is OK if the parts of the distribution you're discarding are the garbage ones.)