LAVA NAT: A Non-Autoregressive Translation Model with Look-Around Decoding and Vocabulary Attention
Table 1: The performances of non-autoregressive methods on WMT14 En↔De, WMT16 Ro→En and IWSLT14 De→En datasets in terms of BLEU score . We use ”kd” to denote ”knowledge distillation”. ”best” represents the highest BLEU score reported in the original paper.
['[ITALIC]  [BOLD] Autoregressive Models', '[BOLD] WMT14 En→De', '[BOLD] WMT14 De→En', '[BOLD] WMT16 Ro→En', '[BOLD] IWSLT14 De→En', '[BOLD] latency', '[BOLD] speedup ']
For all our tasks, we obtain results comparable with Transformer, the state-of-the-art autoregressive model. Our best model achieves 28.42 (+1.1 gain over Transformer), 31.59 (+0.3 gain) and 33.59 (+0.3 gain) BLEU score on WMT14 En→De, WMT14 De→En and IWSLT14 De→En, respectively. More importantly, our LAVA-NAT model decodes much faster than Transformer, which is a big improvement regarding the speed-accuracy trade-off in AT and NAT models. On WMT14 En→De, rescoring 10 candidates leads to a gain of +1.6 BLEU, and rescoring 20 candidates gives about a +2 BLEU score increase. The same phenomenon can be observed on other datasets and for models without knowledge distillation. Nonetheless, rescoring has the weakness of greater time cost, for which dynamic decoding is beneficial to mitigate this issue by speeding up about 3x∼5x comparing to decoding with AT teacher rescoring mechanism. that LAVA NAT gets a nearly 30 times decoding speedup than Transformer, while achieving comparable performances. Compared to other NAT models, we can observe that the LAVA NAT (w/kd) model is almost the fastest (only a little bit behind of Hint-NAT) in terms of latency, and is surprisingly faster than CRF-NAT and PNAT, with better performances. In addition, it’s worth noting that dynamic decoding is vital in speeding up decoding (3∼5 times faster than LAVA NAT with rescoring) while achieving promising results.