Non-autoregressive Machine Translation with Disentangled Context Transformer
Table 6: Dev results from bringing training closer to inference.
['Training Variant', '[BOLD] en\\rightarrowde Step', '[BOLD] en\\rightarrowde BLEU', '[BOLD] ro\\rightarrowen Step', '[BOLD] ro\\rightarrowen BLEU']
Easy-First Training So far we have trained our models to predict every word given a random subset of the other words. But this training scheme yields a gap between training and inference, which might harm the model. We attempt to make training closer to inference by training the DisCo transformer in the easy-first order. Similarly to the inference, we first predict the easy-first order by estimating P(Y_{n}|X) for all n. Then, use that order to determine Y_{{\tt obs}}^{n}. The overall loss will be the sum of the negative loglikelihood of these two steps. In both directions, this easy-first training does not ameliorate performance, suggesting that randomness helps the model. Notice also that the average number of iterations in inference decreases (4.03 vs. 4.29, 2.94 vs. 3.17). The model gets trapped in a sub-optimal solution with reduced iterations due to lack of exploration.