Mastering the Dungeon: Grounded Language Learning by Mechanical Turker Descent
Table 4: Accuracy breakdown by output length (number of actions in sequence y for a given input x).
['Method', 'Length 1', 'Length 2', 'Length 3', 'Length 4']
Agents trained on the collaborative-only baseline data are slightly more effective at learning length 1 examples, but fail to generalize to longer action sequences. For example, Seq2Seq trained by MTD outperforms Seq2Seq trained on the baseline data by up to 180% on length 4 examples. Note that MTD workers are trying to optimize their score which is an average over the entire distribution , so will not be directly aware of or trying to optimize for this breakdown.