Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation
Table 6: Intrinsic evaluation of pronoun gender classification over our new test sets.
['[EMPTY]', '[EMPTY]', 'Masculine P', 'Masculine R', 'Masculine F1', 'Feminine P', 'Feminine R', 'Feminine F1']
We take as our NMT baseline a Transformer Vaswani et al. We set input dimension to 512 and use 6 layers with 8 attention heads and hidden dimension 2048, dropout probabilities of 0.1 and learning rate 2.0. We train this model on WMT’13 Spanish→English data To evaluate how well these NMT systems model pronoun gender, we use them to translate the Spanish text from our new dataset into English. For each target Spanish pronoun, we use an implementation of Vogel et al. All context sentences precede the one containing the pronoun (no lookahead). We initialize our LM experiments from a BERTBASE model (110M parameters; see Devlin et al. To assess how much this model knows about pronoun gender out-of-the-box, we extend the masked LM approach from the original BERT work, inserting a mask token in each dropped subject position (see To generate gender predictions, we deduce masculine if él appears before ella in the list of mask fills and vice versa; we make no gender prediction where neither pronoun appears in the top ten mask fills. We note that the performance we see here is remarkably similar to human ratings of quality for the dataset.