Yea I think those parameters are just cracked compared to what they did for ChatGPT. It's likely a much larger training set with some different training schedule, dropout etc idk. I don't think it's wildly novel though.