Overcoming Statistical Shortcutsfor Open-ended Visual Counting
Table 1: State-of-the-art comparison on a modified version of TallyQA using our Odd-Even-90% strategy. 90% of the even labels and odd labels have been removed from the original training and testing sets respectively. We report the final accuracy on the two testing sets: simple or complex. We also report accuracy on the validation set which follows the training distribution and is used for early-stopping. Parameter counts are in millions.
['[EMPTY]', 'Testing set Simple', 'Testing set Complex', 'Validation set', '# of Parameters']
, we compare our model against state-of-the-art approaches and other reported models on TallyQA. Scores for SCN are averaged over three runs, with a variance for simple and complex of 0.3 and 1.1 respectively. We train and evaluate each model on a modified version of TallyQA using our Odd-Even-90% strategy. Models that over-rely on statistical shortcuts are expected to perform well on its validation set since it follows the training set distribution but suffer from a large loss in accuracy on the testing sets. Interestingly, RCN and Counter reach a high accuracy of 69.53% and 67.50% on the validation set, but suffer from huge losses of -38.89 and -49.51 accuracy points respectively on the simple testing set. We observe similar losses on the complex testing set.