General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping
Table 1: Binomial tests on how different metrics compare in correlation with human judgments. The sign test uses n=sum of positives and negatives; k=number of positives; p=0.5.
['[EMPTY]', '[BOLD] UC ( [ITALIC] nDTW vs) PL', '[BOLD] UC ( [ITALIC] nDTW vs) NE', '[BOLD] UC ( [ITALIC] nDTW vs) ONE', '[BOLD] UC ( [ITALIC] nDTW vs) CLS', '[BOLD] UC ( [ITALIC] nDTW vs) AD', '[BOLD] UC ( [ITALIC] nDTW vs) MD', '[BOLD] SC ( [ITALIC] SDTW vs) SR', '[BOLD] SC ( [ITALIC] SDTW vs) OSR', '[BOLD] SC ( [ITALIC] SDTW vs) SPL', '[BOLD] SC ( [ITALIC] SDTW vs) SED']
To analyze the collected annotations, we first assign nDTW in UC study (similarly SDTW in SC study) a positive/negative sign depending if it has higher/lower correlation than competing metric for a given human ranking of query paths with respect to a reference path, and then compare across all reference paths (discarding ties) using a sign test. Both nDTW and SDTW correlate substantially better with human orderings, compared to the competing metrics in their respective categories.