XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning
Table 3: Percentage of annotated labels in each language agreeing with the majority label. Note that the majority label is highly reliable, as we observed a 100% agreement with the development set labels in the original COPA.
['[EMPTY]', 'et', 'ht', 'id', 'it', 'qu', 'sw', 'ta', 'th', 'tr', 'vi', 'zh']
From the 11 sets of annotation labels we obtain the majority labels (i.e., 6+ translators agree). We observe perfect agreement between our majority labels and the English COPA labels for development data. The small discrepancies in label choices in our work stem not only from the actual semantic ambiguity of the original English question, but also reflect the translators’ different cultural frames of reference and patterns of association. On average, 2.1% of labels in the validation set and 2.4% of labels in the test set do not match the majority label.