Pre-training via Paraphrasing
None
['[EMPTY]', 'IWSLT2017 ar', 'IWSLT2017 de', 'IWSLT2017 fr', 'IWSLT2017 ja', 'IWSLT2017 zh', 'WMT19 de']
Performance varies considerably by language, but reaches 35.8 for German to English, which is the highest score we are aware of for system trained with no bitext. Performance is also strong for some languages using different scripts, such as Arabic to English. However, some languages work less well, notably Japanese. Generating non-English languages proves harder in all cases, particularly those with non-Latin alphabets, but English to French works well. Future work should explore up-sampling rarer languages during pre-training.