One confusing aspect of the ChatGPT release is theres no information about inference. Clearly its not greedy decoding. I wouldnt be surprised if there is a reranking component and/or dynamically set hyperparams.