Non-autoregressive Transformer by Position Learning
Table 4: Results on test set of Quora datasets. “RR”: remove repeats.
['Model', 'Paraphrase(BLEU) w/ RR', 'Paraphrase(BLEU) w/o RR', 'ΔBLEU']
NAT is less effective to distinguish adjacent decoder hidden states, which is copied from the adjacent source representation. To further study this problem, we proposed to evaluate the gains of simply remove the repeated tokens. Removing repeated tokens has little impact for PNAT model, with only 0.05 BLEU differences. However for the NAT-base model, the gap comes with almost 1 BLEU (0.89). The results clearly demonstrate that the explicitly position model essentially learns the sequential information for sequence generation.