Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
Table 1: Generation quality (BLEU↑) and decoding efficiency (tokens/sec↑ for translation, images/sec↑ for image captioning). Decoding efficiency is measured sentence-by-sentence. AR: autoregressive models. b: beam width. idec: the number of refinement steps taken during decoding. Adaptive: the adaptive number of refinement steps. NAT: non-autoregressive transformer models (Gu et al., 2017). FT: fertility. NPD reranking using 100 samples.
['[EMPTY]', '[EMPTY]', 'IWSLT’16 En-De En→', 'IWSLT’16 En-De De→', 'IWSLT’16 En-De GPU', 'IWSLT’16 En-De CPU', 'WMT’16 En-Ro En→', 'WMT’16 En-Ro Ro→', 'WMT’16 En-Ro GPU', 'WMT’16 En-Ro CPU', 'WMT’14 En-De En→', 'WMT’14 En-De De→', 'WMT’14 En-De GPU', 'WMT’14 En-De CPU', 'MS COCO BLEU', 'MS COCO GPU', 'MS COCO CPU']
First, the generation quality improves across all the tasks as we run more refinement steps idec even beyond that used in training (itrain=4), which supports our interpretation as a conditional denoising autoencoder in Sec. To further verify this, we run decoding on WMT’14 (both directions) up to 100 iterations. As shown in Fig.