Overcoming Statistical Shortcutsfor Open-ended Visual Counting
Table 6: Comparison against additional baselines on a modified version of TallyQA using our Odd-Even-90% strategy. 90% of the even labels and odd labels have been removed from the original training and testing sets respectively. We report the final accuracy on the two testing sets: simple or complex. We also report accuracy on the validation set which follows the training distribution and is used for early-stopping.
['[EMPTY]', 'Testing set Simple', 'Testing set Complex', 'Validation set']
S6SS4SSS0Px1 Baselines The Random (train) baseline consists in randomly sampling the predicted count labels following their distribution in the training set. Random (test) samples count labels following their distribution in the testing set. While this baseline leverages knowledge about the testing set distribution, SCN reaches significantly better accuracy scores with +16.88 accuracy points in simple questions and +4.76 against it. The RCN + Uni. Sample. baseline is trained with a different triplets sampling strategy than RCN based on a uniform sampling of the count labels. By doing so, we seek to train a more robust RCN model against shift in distributions between training and testing sets. As expected, we report lower accuracy scores than RCN on the validation set with a loss of -4.09 point. Interestingly, we report lower scores on the simple testing set with a loss of -4.14 points and slightly better score on the complex testing set with a gain of +1.09. Overall, we report significantly lower scores than our SCN with -21.72 accuracy point on simple. These results show that the uniform sampling strategy is not suited to learn robust counting mechanisms.