Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation
Table 9: BLEU score and pronoun accuracy by gender on the WMT’13 Spanish-to-English test set.
['Model', 'Translation Input', 'Classifier Input', 'BLEU', 'Masculine P', 'Masculine R', 'Masculine F1', 'Feminine P', 'Feminine R', 'Feminine F1']
Masculine performance is uniformly strong and our new model, Baseline MT + Gender Tags, gives the best performance on feminine pronouns, achieving a 8.8% F1 score improvement compared to Baseline MT. As for the contextual MT models analyzed in the previous section, the improvement is from addressing the low recall over feminine pronouns in sentence-level translation (while retaining precision). We manually reviewed error cases and noted the major class was from domain shift: where antecedents were close to pronouns in Wikipedia, there were more frequent cases in WMT of antecedents being outside the BERT context window, and these were misclassified. Future work could look at methods for learning pre-trained language models from extended document context.