Transferable Representation Learning in Vision-and-Language Navigation
Table 6: Ablations showing the effect of adapting (or not) the learned representations in each branch of our RCM agent on Validation Seen and Validation Unseen. SPL and SR are reported as percentages and NE and PL in meters.
['Image encoder', 'Language encoder', '[BOLD] Validation Seen PL', '[BOLD] Validation Seen NE ↓', '[BOLD] Validation Seen SR ↑', '[BOLD] Validation Seen SPL ↑', '[BOLD] Validation Unseen PL', '[BOLD] Validation Unseen NE ↓', '[BOLD] Validation Unseen SR ↑', '[BOLD] Validation Unseen SPL ↑']
The second ablation analyzes the effect of transferring representations to either of the language and visual encoders. The learned representations help the agent to generalize on previously unseen environments. When either of the encoders is warm-started, the agent outperforms the baseline success rates and SPL on validation unseen dataset. In the absence of learned representations, the agent overfits on seen environments and as a result the performance improves on the validation seen dataset. Among the agents that have at least one of the encoders warm-started, the agent with both encoders warm-started has significantly higher SPL (7%+) on the validation unseen dataset.