meProp: Sparsified Back Propagation for Accelerated Deep Learningwith Reduced Overfitting
Table 1: Results based on LSTM/MLP models and AdaGrad/Adam optimizers. Time means averaged time per iteration. Iter means the number of iterations to reach the optimal score on development data. The model of this iteration is then used to obtain the test score.
['[BOLD] POS-Tag (AdaGrad)', 'Iter', 'Backprop time (s)', 'Dev Acc (%)', 'Test Acc (%)']
In the table, meProp means applying meProp to the corresponding baseline model, h=500 means that the hidden layer dimension is 500, and k=20 means that meProp uses top-20 elements (among 500 in total) for back propagation. Note that, for fair comparisons, all experiments are first conducted on the development data and the test data is not observable. Then, the optimal number of iterations is decided based on the optimal score on development data, and the model of this iteration is used upon the test data to obtain the test scores. For example, when k=5, it means that the backprop ratio is 5/500=1%. The optimizer is Adam. As we can see, meProp achieves consistently better accuracy than the baseline.