General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping
Table 2: Evaluation metrics as percentages on R2R and R4R Validation Unseen sets for agents with different reward functions. In all metrics, higher means better.
['Agent', '[BOLD] R2R SR', '[BOLD] R2R SPL', '[BOLD] R2R SED', '[BOLD] R2R CLS', '[BOLD] R2R nDTW', '[BOLD] R2R SDTW', '[BOLD] R4R SR', '[BOLD] R4R SPL', '[BOLD] R4R SED', '[BOLD] R4R CLS', '[BOLD] R4R nDTW', '[BOLD] R4R SDTW']
Compared to a goal-oriented reward strategy, taking advantage of nDTW as a reward signal not only results in better performance on nDTW and SDTW metrics but also better performance on prior metrics like CLS and SPL. nDTW shows better differentiation compared to CLS on R4R between goal and fidelity oriented agents. SED scores random paths more highly than those of trained agents, and neither SR nor SED differentiate between goal and fidelity orientation. SPL appears to do so (15.0 vs 21.4), but this is only due to the fact that the fidelity-oriented agent produces paths that have more similar length to the reference paths rather than fidelity to them. As such, SDTW provides the clearest signal for indicating both success and fidelity.