Learning by Asking Questions
Table 3: Accuracy on CLEVR validation data at different budgets B as a function of the bootstrap set size, |Binit|.
['|Binit|', '[BOLD] Budget  [ITALIC] B 0k', '[BOLD] Budget  [ITALIC] B 70k', '[BOLD] Budget  [ITALIC] B 140k', '[BOLD] Budget  [ITALIC] B 210k', '[BOLD] Budget  [ITALIC] B 350k', '[BOLD] Budget  [ITALIC] B 560k', '[BOLD] Budget  [ITALIC] B 630k']
We vary the size of the bootstrap set Binit used for initializing the g,r,v models and analyze its effect on the LBA generated data. A smaller bootstrap set results in reduced performance. We also see that with less than 5% (rows 1 and 2) of the CLEVR training dataset as our bootstrap set, LBA asks questions that can match the performance using the entire CLEVR training set. Empirically, we observed that the generator g performs well on smaller bootstrap sets. However, the relevance model r needs enough valid and invalid (permuted) ( I,q,a) tuples in the bootstrap set to filter irrelevant question proposals. As a result, a smaller bootstrap set affects the sample efficiency of LBA.