Embodied Multimodal Multitask Learning
Table 2: Accuracy of all models on SGN & EQA test sets for both Easy & Hard difficulties.
['[EMPTY]', '[ITALIC] Easy No Aux', '[ITALIC] Easy No Aux', '[ITALIC] Easy Aux', '[ITALIC] Easy Aux', '[ITALIC] Hard No Aux', '[ITALIC] Hard No Aux', '[ITALIC] Hard Aux', '[ITALIC] Hard Aux']
We observe that the Dual-Attention model and many baselines achieve nearly 100% accuracy during training in the Easy setting as seen on the left in Figures Even without auxiliary tasks, Dual-Attention model achieves an accuracy of 86% for SGN and 53% for EQA, whereas the best baseline performance is 33% for both SGN and EQA. Performance of all the baselines is worse than ‘Text only’ model on EQA test set, although the training accuracy is close to 100%. This indicates that baselines tend to overfit on the training set and fail to generalize to questions which contain words never seen in training questions. For the Hard setting, the Dual-Attention model achieves a higher training (87% vs. 70%) as well as test performance (82% vs. 39% for SGN, 59% vs. 33% for EQA with Aux) than the baselines (as seen on the right in These results confirm the hypothesis that prior models, which are designed for a single task, lack the ability to align the words in both the tasks and transfer knowledge across tasks. Lower test accuracy on EQA for most models