CompiLIG at SemEval-2017 Task 1: Cross-Language Plagiarism Detection Methods for Semantic Textual Similarity
Table 3: Results of our submitted systems scored on our 120 annotated pairs and on the same 120 SemEval annotated pairs.
['[BOLD] Methods', '[BOLD] SNLI (4a)', '[BOLD] WMT (4b)', '[BOLD] Mean']
To investigate deeper on this issue, we manually annotated 60 random pairs of each sub-corpus (120 annotated pairs among 500). These annotations provide a second annotator reference. However, the difference in correlation is huge between our annotations and SemEval gold standard on the WMT corpus (4b): 30% on average. The Pearson correlation between our annotated pairs and the related gold standard is 85.76% for the SNLI corpus and 29.16% for the WMT corpus. These results question the validity of the WMT corpus (4b) for semantic textual similarity detection.