I love this explainer by on how human feedback shaped ChatGPT. But how were these human annotators chosen? Do they represent me? Even so, preferences are highly heterogeneous. The average preference is nobodys preference.