FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
Table 2: BLEU scores on two WMT datasets of models using advanced decoding methods. The first block are autoregressive Transformer-base (Vaswani et al., 2017). The second and third blocks are results of models trained w/w.o. knowledge distillation, respectively. n=l×r is the total number of rescoring candidates.
['[BOLD] Models', '[BOLD] WMT2014  [BOLD] EN-DE', '[BOLD] WMT2014  [BOLD] DE-EN', '[BOLD] WMT2016  [BOLD] EN-RO', '[BOLD] WMT2016  [BOLD] RO-EN']
We vary the temperature within {0.1,0.2,0.3,0.4,0.5,1.0} and select the best temperature based on the performance on development sets. For FlowSeq, NPD obtains better results than IWD, showing that FlowSeq still falls behind the autoregressive Transformer on modeling the distributions of target languages. Compared with CMLM ( Ghazvininejad et al., with 10 iterations of refinement, which is a contemporaneous work that achieves state-of-the-art translation performance, FlowSeq obtains competitive performance on both WMT2014 and WMT2016 corpora, with only slight degradation in translation quality. Notably we did not attempt to perform iterative refinement, but there is nothing that makes FlowSeq inherently incompatible with refinement – we leave connecting the two techniques to future work.