Speaker-Follower Models for Vision-and-Language Navigation
Table C.1: Effects of speaker scoring and implementation details in our model. NE is navigation error (in meters); lower is better. SR and OSR are success rate and oracle success rate (%); higher is better. Comparison between the 1st and the 2nd row shows that incorporating speaker scoring is crucial to the performance, matching our intuition in Sec. 3.2 of the main paper of the importance of pragmatic inference. Comparison between the 1st and the 3rd row indicates that state-factored search (Sec. B) produces better results than (standard) beam search on val-unseen. Difference between the 1st and the 4th row shows that using GloVe embeddings pennington2014glove gives slightly higher success rate.
['#', '[EMPTY]', 'Validation-Seen NE ↓', 'Validation-Seen SR ↑', 'Validation-Seen OSR ↑', 'Validation-Unseen NE ↓', 'Validation-Unseen SR ↑', 'Validation-Unseen OSR ↑']
We further study how varying the speaker weight λ in pragmatic inference (Sec. 3.2 of the main paper) influences the final navigation performance. λ=0.95) in Row 1 against using only the follower model to score routes (λ=0) in Row 2, a baseline that still includes search but does not include speaker scores. The large gap in success rate between λ=0.95 and λ=0 shows that including speaker scores is crucial to the performance, confirming the importance of pragmatic inference. In addition, we study how the number of candidate routes, K, used in pragmatic inference impacts the final navigation success rate. The results show that having more routes to choose between leads to better generalization to the unseen new environments (val unseen), but the gain from increasing the number of candidates tends to saturate quickly. We also analyze some implementation details in our model. When generating the predictions for the challenge submission, we modified the method for generating final routes to comply with the challenge guidelines. Hence, our reported navigation error, success rate and oracle success rate are all computed by choosing a single candidate route per instruction. However, the challenge guidelines require that the submitted trajectories must be generated from a single independent evaluation run, where the agent must move sequentially and all its movements during inference must be recorded. So just returning the route selected by pragmatic inference, or by search, would violate the contest guidelines, as this route may not contain all world states explored during search. On the other hand, the agent is allowed to backtrack to a previous location on its path, as long as the agent does not teleport and all its movements are recorded in the final trajectory (which we confirmed with the challenge organizer).