Transferable Representation Learning in Vision-and-Language Navigation
Table 5: Ablations on R2R Validation Seen and Validation Unseen sets, showing results in VLN for different combinations of pre-training tasks. SPL and SR are reported as percentages and NE and PL in meters.
['Method', 'cma', 'nvs', '[BOLD] Validation Seen PL', '[BOLD] Validation Seen NE ↓', '[BOLD] Validation Seen SR ↑', '[BOLD] Validation Seen SPL ↑', '[BOLD] Validation Unseen PL', '[BOLD] Validation Unseen NE ↓', '[BOLD] Validation Unseen SR ↑', '[BOLD] Validation Unseen SPL ↑']
The first ablation study analyzes the effectiveness of each task individually in learning representations that can benefit the navigation agent. When pre-trainning CMA and NVS jointly, we see a consistent 11-12% improvement in SR for both the SF and RCM agents as well as improvement in agent’s path length, thereby also improving SPL. When pre-training CMA only, we see a consistent 8-9% improvement in SR for both the SF and RCM agents. When pre-training NVS only, we see a drop in performance. Since there are no cross-modal components to train the language encoder in NVS, training on NVS alone fails to provide a good initialization point for the downstream navigation task that requires cross-modal associations. However, pre-training with NVS and CMA jointly affords the model additional opportunities to improve visual-only pre-training (due to NVS), without compromising cross-modal alignment (due to CMA).