Embodied Multimodal Multitask Learning
Table 4: The performance of a trained policy appended with object detectors on instructions containing unseen words (‘red’ and ‘pillar’).
['Model', 'No Aux SGN', 'No Aux EQA', 'Aux SGN', 'Aux EQA']
The results indicate that SA and GA1 contribute the most to the performance of the Dual-Attention model. This is expected as GA2 is designed to attend to different objects and attributes based on the question and is used mainly for answer prediction. It is not critical for SGN as the spatial attention map consists of locations of relevant objects, which is sufficient for navigating to the correct object. Dual-Attention models trained only on single tasks work well on SGN, especially with auxiliary tasks. This is because the auxiliary task for single task models includes object detection labels corresponding to the words in the test set. This highlights a key advantage of the proposed model. Due to its modular and interpretable design, the model can be used for transferring the policy to new objects and attributes without fine-tuning as discussed in the following subsection. In contrast, our model can be used for transfer to new words by training an object detector for each new word and appending it to the image representation xI. In order to test this, we train a single-task SGN model using the proposed architecture on the training set for instructions. We use auxiliary tasks but only for words in the vocabulary of the instructions training set. After training the policy, we would like the agent to follow instructions containing test words ‘red’ and ‘pillar’, which the agent has never seen or received any supervision about how this attribute or object looks visually. For transferring the policy, we assume access to two object detectors which would give object detections for ‘red’ and ‘pillar’ separately. We resize the object detections to the size of a feature map in the image representation (H×W) and append them as channels to the image representation. We also append the words ‘red’ and ‘pillar’ to the bag-of-words representations in the same order such that they are aligned with the appended feature maps. We randomly initialize the embeddings of the new words for computing the sentence embedding. This suggests that a trained policy can be scaled to more objects provided the complexity of navigation remains consistent.