PPO was used to align(optimize) the LLM (policy). This policy was initialized with pretrained(SSL) GPT3. The reward model and even value function of PPO was init with pretrained (SSL) GPT3. Thus SSL is a big part of InstructGPT a