Gender in Danger?Evaluating Speech Translation Technology on the MuST-SHE Corpus
Table 3: BLEU scores for En-It and En-Fr on MuST-SHE. Results are provided for the whole dataset (All) as well as split according to feminine and masculine word forms. Results are calculated for both the Correct and Wrong datasets, and their difference is provided (Diff).
['[EMPTY]', 'Systems', '[BOLD] All Correct', '[BOLD] All Wrong', '[BOLD] All Diff', '[BOLD] Feminine Correct', '[BOLD] Feminine Wrong', '[BOLD] Feminine Diff', '[BOLD] Masculine Correct', '[BOLD] Masculine Wrong', '[BOLD] Masculine Diff']
BLEU. Looking at overall translation quality (All/Correct column), the results on both language pairs show that the highest performance is achieved by cascade architectures, which are better than End2End by 2.6 points for En-It and 4.3 for En-Fr. We do not observe a statistically significant difference between Cascade and Cascade+Tag, suggesting that the injection of gender information into Cascade+Tag does not have visible effects in terms of translation quality, even on a focused dataset like MuST-SHE where each segment contains at least one gender realization.