Why do you say they are similar? Interpretable Semantic Textual Similarity
Table 3: Inter-tagger agreement on the Headlines and Images dataset.
['[EMPTY]', 'Headlines', 'Images']
To measure the viability and quality of the annotation and to calculate the inter-tagger agreement (ITA), two annotators annotated, individually, a random subset of 20 sentence pairs, 10 from each dataset. Both annotators previously read the guidelines and agreed on mutual uncertainties. The agreement was computed using the evaluation script, where one tagger was taken as the system and the other one as the gold standard. Notice that the metrics used to compute the ITA are the same as the metrics used to evaluate system performance on the task.