Not exactly. While you could get the prob of the generated sentence, even if the model is well calibrated, that's not the same as measuring the probability of being correct. Something similar to the reward model