Pushing the bounds of dropout
Table 3: PTB training and validation XEs for AMC at λ∈{0,0.8,1} per word frequency. Note how DET dominates AMC on the training set, but AMC is better for rare words in the validation set.
['frequency', 'num targets', 'training DET', 'training ×0.8', 'training AMC', 'validation DET', 'validation ×0.8', 'validation AMC']
On the training set, the gap between deterministic dropout and AMC is wider for low frequency words. On the validation set, AMC is worse for frequent words but better for rare words. The ×0.8 dropout multiplier just finds a reasonable compromise.