Learning to Answer Questions From Image Using Convolutional Neural Network
Table 3: Image QA performances on COCO-QA.
['[EMPTY]', 'Accuracy', 'WUPS @0.9', 'WUPS @0.0']
For DAQUAR-All and DAQUAR-Reduced datasets with multiple words as the answer to the question, we treat the answer comprising multiple words as an individual class for training and testing. For the DAQUAR-Reduced dataset, besides the Neural-Image-QA approach, the VSE model is also compared on “single word”. GUESS is the model which randomly outputs the answer according to the question type. BOW treats each word of the question equally and sums all the word vectors to predict the answer by logistic regression. IMG+BOW performs the multinomial logistic regression based on the image feature and a BOW vector obtained by summing all the word vectors of the question. VIS+LSTM and 2-VIS+BLSTM are two versions of the VSE model. VIS+LSTM has only a single LSTM to encode the image and question in one direction, while 2-VIS+BLSTM uses a bidirectional LSTM to encode the image and question along with both directions to fully exploit the interactions between image and each word of the question. It can be observed that 2-VIS+BLSTM outperforms VIS+LSTM with a big margin. Our proposed CNN model significantly outperforms the competitor models. More specifically, for the case of “single word”, our proposed CNN achieves nearly 20% improvement in terms of accuracy over the best competitor model 2-VIS+BLSTM. For the COCO-QA dataset, IMG+BOW outperforms VIS+LSTM and 2-VIS+BLSTM, demonstrating that the simple multinomial logistic regression of IMG+BOW can better model the interactions between image and question, compared with the LSTMs of VIS+LSTM and 2-VIS+BLSTM. By averaging VIS+LSTM, 2-VIS+BLSTM, and IMG+BOW, the FULL model is developed, which summarizes the interactions between image and question from different perspectives thus yields a much better performance. The reason may be that the image representation is of highly semantic meaning, which should interact with the high semantic components of the question. Our CNN model firstly uses the convolutional architectures to compose the words to highly semantic representations. Afterwards, we let the image meet the composed highly semantic representations and use convolutional architectures to exploit their relations and interactions for the answer prediction. As such, Our CNN model can well model the relations between image and question, and thus obtain the best performances.