A Comparative Study on Regularization Strategiesfor Embedding-based Neural Networks
Table 1: Accuracy in percentage when we combine ℓ2-norm of weights and embeddings (Experiment I). Bold numbers are among highest accuracies (greater than peak performance minus 1.5 times standard deviation, i.e., 1.26 in percentage).
['[ITALIC] λembed', '[ITALIC] λW 0', '[ITALIC] λW 10−4', '[ITALIC] λW 3⋅10−4', '[ITALIC] λW 10−3']
In a certain range of coefficients, weights and embeddings are complementary: given one hyperparameter, we can tune the other to achieve a result among highest ones. Such compensation is also observed in penalizing ℓ2-norm versus dropout the peak performance is obtained by pure ℓ2 regularization, applying dropout with small ℓ2 penalty also achieves a similar accuracy. The dropout rate is not very sensitive, provided it is small.