Why do you say they are similar? Interpretable Semantic Textual Similarity
Table 6: Results (F1) of our three systems on each of the datasets. Columns show the results on each evaluation criteria, where T+S stands for “Type and Score”. Best results in bold. The last row shows the results for the Full system when using gold standard chunks instead of automatically produced chunks.
['[EMPTY]', '[BOLD] Headlines ALI', '[BOLD] Headlines TYPE', '[BOLD] Headlines SCORE', '[BOLD] Headlines T+S', '[BOLD] Images ALI', '[BOLD] Images TYPE', '[BOLD] Images SCORE', '[BOLD] Images T+S']
The better chunking and alignment (Base+ and Full) improves the alignment F1 score more than 10 points in both datasets with respect to Base. The poor performance in alignment causes the baseline system to also attain low F1 scores in type and score, as well as the overall F1 score (T+S). The comparison between Base+ and Full shows that the classifier is able to better assign types, specially for Images. The method to produce the score is also stronger in Full, and thus produces the best overall F1 (T+S). Note that the performance for the four available metrics decreases, as all metrics are bounded by ALI, and T+S is bounded by both TYPE and SCORE. We performed an analysis of the errors performed by the Full system at each level of processing, starting with chunking. The results improve for both datasets, with very high alignment results, and show that chunking quality is key for good performance. We can thus conclude that Headlines is more difficult to chunk than Images, which causes worse performance on this dataset. Some of the errors in chunking seem to be related to verbs, as shown in an example below, and could be caused by the particular syntactic structures used in news headlines, which are different from those expected by the automatic chunker. * 