Pre-trained Language Model Representations for Language Generation
Table 1: BLEU on newstest2018 of WMT English-German in three simulated bitext size scenarios.
['[EMPTY]', '160K', '640K', '5186K']
We further improve src-elmo by sharing learned word representations in the decoder by tying input and output embeddings (src-elmo+shdemb). This configuration performs even better than src-elmo with a gain of 5.3 BLEU in the 160K setup. Sharing decoder embeddings is equally applicable to src-ft. Language model representations are much less effective in the decoder: tgt-ft improves the 160K bitext setup but yields no improvements thereafter and tgt-elmo performs even worse. We conjecture that pre-trained representations give much easier wins in the encoder.