Object-and-Action Aware Model for Visual Language Navigation
Table 2: Comparison of object-aware module (OA) and action-aware module (AA) on the R2R dataset. ‘w/o’ denotes to remove the module from the full model. The AA module contributes more than the OA module.
['[BOLD] Model', '[BOLD] Val Seen SR↑', '[BOLD] Val Seen OSR↑', '[BOLD] Val Seen SDTW↑', '[BOLD] Val Seen SPL↑', '[BOLD] Val Seen TL↓', '[BOLD] Val Seen CLS↑', '[BOLD] Val Unseen SR↑', '[BOLD] Val Unseen OSR↑', '[BOLD] Val Unseen SDTW↑', '[BOLD] Val Unseen SPL↑', '[BOLD] Val Unseen TL↓', '[BOLD] Val Unseen CLS↑']
We further study the importance of the object-aware (OA) module and the action-aware (AA) module within the trained OAAM. There is about 20% performance drop in SR when OA or AA works alone. Furthermore, the AA module performs consistently better than the object-aware module on ValSeen, ValUnSeen, and Test splits, in terms of SPL, SR, or OSR. For example, when AA module is removed, the SR is about 20% lower than that when OA module is removed on the Unseen split. Fortunately, when OA and AA are adaptively combined, a much better performance is achieved (10%∼28% improvement in SPL on the Val Unseen split). Additionally, we calculate the mean of wt of all steps on ValSeen and ValUnseen. Results show that OA and AA are asigned 0.18 and 0.82 on average, respectively. This also reflects AA contributes more to the final results. Lastly, we make a try to conduct quantitative evaluation of these two modules based on NLTK tags, although it is not perfect. An attention is considered success if the top3 words attended by OA contain nouns (verbs for AA). OA and AA achieve accuracy of about 80% and 75% on average, respectively. This at some extent indicates improvement room for these modules.