A negative case analysis of visual grounding methods for VQA
Table 2: p-values from the Welch’s t-tests and the percentage of overlap between the predictions (Ovp.) of different variants of HINT and SCR.
['Methods', '[ITALIC] p', 'Ovp.(%)']
We pick Welch’s t-test over the Student’s t-test, because the latter assumes equal variances for predictions from different variants. To perform the tests, we first randomly sample 5000 subsets of non-overlapping test instances. We then average the accuracy of each subset across 5 runs, obtaining 5000 values. Next, we run the t-tests for HINT and SCR separately on the subset accuracies. Using a confidence level of 95% (α=0.05), we fail to reject the null hypothesis that the mean difference between the paired values is 0, showing that the variants are not statistically significantly different from each other. We also compare the predictions of HINT/SCR against baseline, and find that p-values are all zeros, showing that the differences have statistical significance. Percentage of Overlaps: To further check if the variants trained on irrelevant or random regions gain performance in a manner similar to the models trained on relevant regions , we compute the overlap between their predictions on VQA-CPv2’s test set. The percentage of overlap is defined as: % Overlap =nsamentotal×100%, where, nsame denotes the number of instances where either both variants were correct or both were incorrect and ntotal denotes the total number of test instances. We find 89.7−91.9% and 89.5−92.0% overlaps for different variants of HINT and SCR respectively. These high overlaps suggest that the variants are not working in fundamentally different manners.