Learning Robust Representations of Text
Table 1: Accuracy (%) with increasing word-level dropout across the four datasets. For each dataset, we apply four levels of noise α={0,0.1,0.2,0.3}; the best result for each combination of α and dataset is indicated in bold. The Baseline model is a simple CNN model without regularization. The last model combines dropout and our method with fixed parameters β and λ as indicated.
['Dataset Word dropout rate ( [ITALIC] α)', '[EMPTY]', 'MR 0', 'MR 0.1', 'MR 0.2', 'MR 0.3', 'Subj 0', 'Subj 0.1', 'Subj 0.2', 'Subj 0.3']
In general, increasing the word-level dropout noise leads to a drop in accuracy for all four datasets, however the relative dropoff in accuracy for Robust Regularization is less than for Word Dropout, and in 15 out of 16 cases (four noise levels across the four datasets), our method achieves the best result. Note that this includes the case of α=0, where the test data is left in its original form, which shows that Robust Regularization is also an effective means of preventing overfitting in the model.