Learning Visual Question Answering by Bootstrapping Hard Attention
Table 4: Results, in %, on CLEVR. SAN denotes the SAN [9] implementation of [34]. SAN* denotes the SAN implementation of [25]. Object RN** [55] and Stack-NMNs** [52] report the results only on the validation set, whereas others report on the test set. Overall performance of Stack-NMNs** [52] is measured with the “expert layout” (similar to N2NMN) yielding 96.6 and without it (93.0). DDRprog− [71], PG+EE (700k)− [70], TbD−, and TbD+hres− [50] are trained with a privileged state-description, while others are trained directly from images-questions-answers. TbD+hres [50] uses high-resolution (28x28) spatial tensor, while majority uses either 8x8 or 14x14. HAN+Sum/RN+ denotes a larger relational model, or a different hyper-parameters setup, than the model of [25]. HAN+RN++ denotes HAN+RN+ with larger input images with spatial dimensions 224x224 as opposed to 128x128, and larger image tensors with spatial dimension 14x14 as opposed to 8x8.
['Model', '[BOLD] Overall', 'Count', 'Exist', 'Compare Numbers', 'Query Attribute', 'Compare Attribute']
(TbD+hres) have noted increasing the spatial resolution definitely helps in achieving better performance. Finally, through a visual inspection, we have observed that the fraction of input cells that we have experimented with (k=16 for 8x8 spatial tensor, and k=64 for 14x14 spatial tensor) is sufficient to cover all the important objects in the image, and thus the mechanism resembles more the saliency mechanism. It is worth noting, the hard-attention mechanism often selects a few cells that correspond to the object as this is sufficient to recognize the object’s properties such as size, material, color, type, and spatial location.