Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Table 2: F1 score (the higher the better) on the validation set of the parsing task.
['Approach', 'F1']
This different operating regime makes for an interesting comparison, as one would not expect the baseline algorithm to make many mistakes. In this table we report the F1 score on the WSJ 22 development set [hovy-EtAl:2006:NAACL]. We should also emphasize that there are only 40k training instances, so overfitting contributes largely to the performance of our system. Whether the effect of sampling during training helps with regard to overfitting or the training/inference mismatch is unclear, but the result is positive and additive with dropout. Once again, a model trained by always sampling from itself instead of using the groundtruth previous token as input yielded very bad results, in fact so bad that the resulting trees were often not valid trees ( hence the “-” in the corresponding F1 metric).