First finetuned to browse the web from human demonstrations. Then uses RLHF similar to ChatGPT to favor answers that are deemed to be more truthful and useful. As said, ChatGPT seems very much inspired by WebGPT.