Unified Pragmatic Models for Generating and Following Instructions
Table 2: Instruction-following results in the SCONE domains. The table shows accuracy on the test set. For reference, we also show prior results from Guu et al. (2017) (GPLL), although our models use more supervision at training time.
['listener', 'Alchemy', 'Scene', 'Tangrams']
We see gains from the rational system L0⋅L1 in both the Alchemy and Scene domains. The pragmatic inference procedure allows correcting errors or overly-literal interpretations from the base listener. The base listener (left) interprets then to orange’s other side incorrectly, while the rational listener discounts this interpretation (it could, for example, be better described by to the left of blue) and produces the action the descriptions were meant to describe (right). To the extent that human annotators already account for pragmatic effects when generating instructions, examples like these suggest that our model’s explicit reasoning is able to capture interpretation behavior that the base sequence-to-sequence listener model is unable to model.