I think part of the problem is that it's really hard to talk about how stuff like this works without using metaphors of sentience - we say "so ChatGPT sees that words like this tend to be followed with words like that, and then it decides to-"- but it doesn't