To generate a response, ChatGPT takes an input prompt (e.g. a message from a user) and processes it through its internal layers to generate a probability distribution over the possible responses. It then uses this distribution to sample a response which is returned to the user.