meProp: Sparsified Back Propagation for Accelerated Deep Learningwith Reduced Overfitting
Table 3: Results based on the same k and h.
['[BOLD] POS-Tag (Adam)', 'Iter', 'Test Acc (%)']
We still have a question: does the top-k meProp work well simply because the original model does not require that big dimension of the hidden layers? For example, the meProp (topk=5) works simply because the LSTM works well with the hidden dimension of 5, and there is no need to use the hidden dimension of 500. As we can see, however, the results of the small hidden dimensions are much worse than those of meProp.