Gender in Danger?Evaluating Speech Translation Technology on the MuST-SHE Corpus
Table 4: Accuracy scores for En-It and En-Fr on MuST-SHE. Results are provided for the whole dataset (All) as well as split according to feminine and masculine word forms. Results are calculated for both the Correct and Wrong datasets, and their difference is provided (Diff).
['[EMPTY]', 'Systems', '[BOLD] All Correct', '[BOLD] All Wrong', '[BOLD] All Diff', '[BOLD] Feminine Correct', '[BOLD] Feminine Wrong', '[BOLD] Feminine Diff', '[BOLD] Masculine Correct', '[BOLD] Masculine Wrong', '[BOLD] Masculine Diff']
Accuracy. While the All/Diff BLEU results for End2End and Cascade were identical on both languages, the All/Diff accuracy scores show that, although End2End performs better than Cascade for En-It, it performs worse for En-Fr. Also, with regards to Cascade+Tag, we can see that the Diff value is higher on the Masculine subset, thus showing that also this system is affected by gender bias, although to a lesser extent.