Evaluating Multimodal Representations on Visual Semantic Textual Similarity
Table 2: Overall item similarity and disagreement of the AMT annotations.
['[EMPTY]', '[BOLD] #Pairs', '[BOLD] Mean', '[BOLD] Median', '[BOLD] STD', '[BOLD] #Zeros']
We defined item disagreement as the standard deviation of the annotated similarity value. The low average similarity can be explained by the high number of zero-similarity pairs. Item disagreement is moderately low (about 0.6 points out of 5) which is in accordance with the high correlation between the annotators.