Pre-training via Paraphrasing
None
['[EMPTY]', '[EMPTY]', 'Target de', 'Target en', 'Target it', 'Target nl', 'Target ro']
Performance varies considerably by language, but reaches 35.8 for German to English, which is the highest score we are aware of for system trained with no bitext. Performance is also strong for some languages using different scripts, such as Arabic to English. However, some languages work less well, notably Japanese. Generating non-English languages proves harder in all cases, particularly those with non-Latin alphabets, but English to French works well. Future work should explore up-sampling rarer languages during pre-training.