MASS: Masked Sequence to Sequence Pre-training for Language Generation
Table 6: The comparison between MASS and the ablation methods in terms of BLEU score on the unsupervised en-fr translation.
['Method  [ITALIC] Discrete', 'BLEU 36.9', 'Method  [ITALIC] Feed', 'BLEU 35.3', 'Method MASS', 'BLEU 37.5']
Ablation Study of MASS In our masked sequence to sequence pre-training, we have two careful designs: (1) We mask consecutive tokens in the encoder side, and thus predict consecutive tokens in the decoder side, which can build better language modeling capability than just predicting discrete tokens. (2) In this section, we conduct two ablation studies to verify the effectiveness of the two designs in MASS. The first study is to randomly mask discrete tokens instead of consecutive tokens in MASS, denoted as Discrete. The second study is to feed all the tokens to the decoder instead of masking the input tokens of the decoder that are not masked in the encoder side, denoted as Feed. It can be seen that both Discrete and Feed perform worse than MASS, demonstrating the effectiveness of the two designs in MASS.