Pushing the bounds of dropout
Table 4: Validation and test perplexities on PTB and Wikitext-2 with various evaluation strategies and default or optimal validation softmax temperatures. Our baseline results correspond to DET at temperature 1. Note that AMC does not benefit from setting the optimal softmax temperature (“opt”), while DET is improved by it almost to the point of matching AMC which supports the smoothing hypothesis.
['[EMPTY]', 'Dataset', 'Temp', 'DET', 'Geometric ( [ITALIC] α=0) ×0.8', 'Geometric ( [ITALIC] α=0) ×0.9', 'Geometric ( [ITALIC] α=0) ×1.0', 'Power  [ITALIC] α=0.5 ×0.8', 'Power  [ITALIC] α=0.5 ×0.9', 'Power  [ITALIC] α=0.5 ×1.0', 'Arithmetic ( [ITALIC] α=1) ×0.8', 'Arithmetic ( [ITALIC] α=1) ×0.9', 'Arithmetic ( [ITALIC] α=1) ×1.0']
To test the hypothesis that AMC’s advantage lies in smoothing, we tested how performing smoothing by other means affects the results. In this experiment, on a trained model the temperature of the final softmax was optimised on the validation set and the model was applied with the optimal temperature to the validation and test sets. In fact, the optimal temperature for AMC with λ=1 was slightly lower than 1, which corresponds to sharpening, not smoothing.