Speaker-Follower Models for Vision-and-Language Navigation
Table 2: Performance comparison of our method to previous work. NE is navigation error (in meters); lower is better. SR and OSR are success rate and oracle success rate (%) respectively (higher is better). Trajectory length (TL) on the test set is reported for completeness. *: When submitting to the Vision-and-Language Navigation Challenge, we modified our search procedure to maintain physical plausibility and to comply with the challenge guidelines. The resulting trajectory has higher oracle success rate while being very long. See Sec. E in the supplementary material for details.
['Method', 'Validation-Seen NE ↓', 'Validation-Seen SR ↑', 'Validation-Seen OSR ↑', 'Validation-Unseen NE ↓', 'Validation-Unseen SR ↑', 'Validation-Unseen OSR ↑', 'Test (unseen) NE ↓', 'Test (unseen) SR ↑', 'Test (unseen) OSR ↑', 'Test (unseen) TL ↓']
In the table, “Random” is randomly picking a direction and going towards that direction for 5 steps. “ours” shows our performance using the route selected by our pragmatic inference procedure, while “ours (challenge participation)” uses a modified inference procedure for submission to the Vision-and-Language Navigation Challenge (See Sec. E in the supplementary material for details). Our method more than doubles the success rate of the state-of-the-art RPA approach, and on the test set achieves a final success rate of 53.5%. This represents a large reduction in the gap between machine and human performance on this task.