Reasoning about Pragmatics with Neural Listeners and Speakers
Table 1: S1 accuracy vs. number of samples.
['# samples', '1', '10', '100', '1000']
Next we turn to the computational efficiency of the reasoning model. As in all sampling-based inference, the number of samples that must be drawn from the proposal is of critical interest—if too many samples are needed, the model will be too slow to use in practice. Having fixed λ=0.02 in the preceding section, we measure accuracy for versions of the reasoning model that draw 1, 10, 100, and 1000 samples. We find that gains continue up to 100 samples.