I think in the case of ChatGPT it is largely in a separate layer based on RLHF But there is nothing they would like better than for it to be hopelessly entangled into the models at every layer, and for the training data itself to be irrev