The Wisdom of MaSSeS:Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA
Table 2: Examples from the validation splits of VQA 1.0 (top) and VizWiz (bottom). For each example, we report the pattern of answers provided by annotators (unique answer: frequency), the prediction of the model, and the scores (note that acm, SeS, MaSSeS are computed using threshold 0.9). Answers that are grouped together by SeS are included in square brackets.
['[BOLD] dataset', '[BOLD] n.', '[BOLD] answers', '[BOLD] prediction', '[BOLD] VQA3+', '[BOLD] acm', '[BOLD] Ma', '[BOLD] S', '[BOLD] SeS', '[BOLD] MaSSeS']
Starting from VQA 1.0, we notice that examples 1 and 2 are considered as 100% correct by both VQA3 + and MaSSeS. The former metric assigns this score because ‘yellow’ and ‘refrigerator’ have frequency equal to or greater than 4. As for MaSSeS, this score is produced because (a) the two answers have max frequency, and (b) the SeS score assigned to the response pattern is the highest (i.e. 1.0) due to their semantic consistency. That is, all the answers are grouped together since their cosine similarity with the centroid is equal or greater than 0.9. Notably, acm produces a similar score in example 2, but very different (i.e., much lower) in example 1, though the words involved are semantically very similar (very similar colors). Moving to example 3, we observe that MaSSeS assigns a lower score (0.67) compared to VQA3+ (1.0) since SeS makes a fine-grained distinction between generic ‘rackets’ and specific ones (i.e., for ‘tennis’). This proves the validity and precision our semantic similarity component, especially in comparison with acm, whose high score does not account for such distinction (0.98). As for example 4, the score output by MaSSeS (1.0) turns out to be higher than both VQA3+ (0.6) and acm (0.7) due to the extremely high semantic consistency of the answers.