Cross-Lingual Vision-Language Navigation
Table 1: Performance comparison for zero-shot setting. Reported results are averages of 5 individual runs. train w/ MT denotes the model trained with Chinese MT data. meta-learner is the framework presented in Figure 1 that aggregates two learning schemes with a cross-lingual meta-learner (Section 4.1). meta+txt2img equips the meta-learner with the txt2img module introduced in Section 4.2. The first three models are all for zero-shot learning. The last one, train w/ AN, is trained with 100% human-annotated Chinese data. All models are tested with human-annotated Chinese instructions.
['Model', '[BOLD] Validation Seen PL', '[BOLD] Validation Seen NE \\downarrow', '[BOLD] Validation Seen SR \\uparrow', '[BOLD] Validation Seen SPL \\uparrow', '[BOLD] Validation Seen CLS \\uparrow', '[BOLD] Validation Unseen PL', '[BOLD] Validation Unseen NE \\downarrow', '[BOLD] Validation Unseen SR \\uparrow', '[BOLD] Validation Unseen SPL \\uparrow', '[BOLD] Validation Unseen CLS \\uparrow']
First, the clear gap between train w/ MT and meta-learner proves that our meta-learner can successfully aggregate cross-representations of both human-annotated and MT data, which also indicates the insufficiency of solely using MT data for zero-shot learning. Second, the txt2img module further improves navigation efficiency (SPL) and instruction following fidelity (CLS), which validates the effectiveness of grounding and aligning cross-lingual representations via the visual space. Moreover, even though the agent does not have access to any annotated target language data, it achieves competitive results compared to train w/ AN that is trained with 100% annotated target language data. For zero-shot learning, our method (meta+txt2img) improves over the model trained with MT data only. For transfer learning, our method can efficiently transfer knowledge between Chinese and English data. The results are coherent with the reported results on the validation set.