NLVR2 Visual Bias Analysis
Table 1: Analysis of image pairs that occur multiple times in the training data with the same label.
['[BOLD] Pair Freq.', '[BOLD] # Pairs', '[BOLD] Expected # Same', '[BOLD] Expected % Same', '[BOLD] Obs. # Same', '[BOLD] Obs. % Same']
Pairs that occur more than twice in the data are more interesting. Consider an event that is less likely: pairs that occur three times with the same label. The probability for a pair to have the same label (true/true/true or false/false/false) is 0.25. There are 345 pairs that occur three times, so we would expect around 43 to have the same label in all instances. However, we observe 128 have the same label. We can generalize this analysis to all observed pair frequencies in the training data.