Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Table 1: Various metrics (the higher the better) on the MSCOCO development set for the image captioning task.
['Approach vs Metric', 'BLEU-4', 'METEOR', 'CIDER']
Each of these metrics is a variant of estimating the overlap between the obtained sequence of words and the target one. Since there were 5 target captions per image, the best result is always chosen. To the best of our knowledge, the baseline results are consistent (slightly better) with the current state-of-the-art on that task. While dropout helped in terms of log likelihood (as expected but not shown), it had a negative impact on the real metrics. On the other hand, scheduled sampling successfully trained a model more resilient to failures due to training and inference mismatch, which likely yielded higher quality captions according to all the metrics. Ensembling models also yielded better performance, both for the baseline and the schedule sampling approach. It is also interesting to note that a model trained while always sampling from itself (hence in a regime similar to inference), dubbed Always Sampling in the table, yielded very poor performance, as expected because the model has a hard time learning the task in that case. We also trained a model with scheduled sampling, but instead of sampling from the model, we sampled from a uniform distribution, in order to verify that it was important to build on the current model and that the performance boost was not just a simple form of regularization. We called this Uniform Scheduled Sampling and the results are better than the baseline, but not as good as our proposed approach. We also experimented with flipping the coin once per sequence instead of once per token, but the results were as poor as the Always Sampling approach.