Sequence Modeling with Unconstrained Generation Order
Table 2: Training strategies of INTRUS. MT task, scores on the WMT En-Ru 2012-2013 test sets.
['Training strategy', 'INTRUS', 'Argmax', 'Pretraining left-to-right', 'No pre- training', 'Only pretraining uniform', 'Only pretraining left-to-right', 'Baseline left-to-right']
In preliminary experiments, we also observed that introducing any of the two pretraining strategies increases the overall robustness of our training procedure and helps to avoid convergence to poor local optima. We attribute this to the fact that a pretrained model provides the main algorithm with a good initial exploration of the trajectory space T∗, while the Argmax training strategy tends to quickly converge to the current best trajectory which may not be globally optimal. This leads to poor performance and unstable results. This is the only strategy that required several consecutive runs to obtain reasonable quality, despite the fact that it starts from a good pretrained model.