The Wisdom of MaSSeS:Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA
Table 1: Results of VQA3+, WUPS-acm, WUPS–mcm, MaSSeS and its components on four VQA datasets.
['[BOLD] dataset', '[BOLD] metric  [BOLD] VQA3+', '[BOLD] metric  [BOLD] WUPS', '[BOLD] metric  [BOLD] WUPS', '[BOLD] metric  [BOLD] MaSSeS', '[BOLD] metric  [BOLD] MaSSeS', '[BOLD] metric  [BOLD] MaSSeS', '[BOLD] metric  [BOLD] MaSSeS', '[BOLD] metric  [BOLD] MaSSeS', '[BOLD] metric  [BOLD] MaSSeS', '[BOLD] metric  [BOLD] MaSSeS']
Note that columns VQA3 + , WUPS-acm, WUPS-mcm, Ma, MaS, and MaSSeS are accuracies, while S and SeS are reliability scores. As can be noted, accuracies obtained with both versions of MaSSeS are generally lower compared to those of VQA3+, with the drop being particularly accentuated for VizWiz. As can be seen, the scores produced by our metric (blue) are ‘distributed’ across the x-axis (from 0 to 1), while those produced by VQA3+ (red) are grouped into 5 ‘classes’. Moreover, we observe that our metric is much more reluctant to output score 1. (recall that if an element is not max it is not considered as 100% correct by Ma). This drop is further accentuated by multiplying Ma by either S (to obtain MaS) or SeS (to obtain MaSSeS). Since the values of these components cannot exceed 1, the resulting score will be lowered according to the degree of subjectivity of the dataset. Bearing this in mind, it is worth focusing on the scores of S and SeS in each dataset. As can be noticed, S in VQA is relatively high, with most of the answers being grouped in the rightmost bars (0.8 or more). In contrast, we observe an almost normal distribution of S in VizWiz, with very few answers being scored with high values. When injecting semantic information into subjectivity (SeS0.9), however, the distribution changes. Indeed, we observe much less cases scored with extremely low values and much many cases with high values. In numbers, this is reflected in an overall increase of 8 points from S (0.46) to SeS (0.54). A similar pattern is also observed in VQA 1.0 (+5 points). It is worth mentioning that using a lowest similarity threshold (0.7) makes the increase between S and SeS even bigger. This, in turn, makes the MaSSeS score significantly higher and comparable to VQA3+ in the three VQA-based datasets (not for VizWiz).