What I think is going on with ChatGPT is that there are few natural trade-offs for some values,whether avoiding slurring or avoiding littering (yes I tried it). So there are few discussions on such tradeoffs for it to learn from.