ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation
Table 5: Test results of non-autoregressive models when training with the references (“baseline”), distilled outputs (“distill”), and energy (“ENGINE”). Oracle lengths are used for decoding. Here, ENGINE uses BiLSTM inference networks and pretrained seq2seq AR energies. ENGINE outperforms training on both the references and a pseudocorpus.
['[EMPTY]', 'IWSLT14 DE-EN Energy (↓)', 'IWSLT14 DE-EN BLEU (↑)', 'WMT16 RO-EN Energy (↓)', 'WMT16 RO-EN BLEU (↑)']
We compared training non-autoregressive models using the references, distilled outputs, and as inference networks on both datasets. The inference networks improve over training with the references by 11.27 BLEU on DE-EN and 12.22 BLEU on RO-EN. In addition, inference networks consistently improve over non-autoregressive networks trained on the distilled outputs. In order to compare ENGINE with training on distilled outputs, we train BiLSTM models in three ways: “baseline” which is trained with the human-written reference translations, “distill” which is trained with the distilled outputs (generated using the autoregressive models), and “ENGINE”, our method which trains the BiLSTM as an inference network to minimize the pretrained seq2seq autoregressive energy. Oracle lengths are used for decoding. Although the results shown here are lower than the transformer results, the trend is clearly indicated.