The values we bake into foundational models will be reinforced and replicated billions of times. With chatGPT these values come partly from web data and from the human raters who were used for preference labelling. Would be very enlightening to see the annotator's instructions.