Leap-LSTM: Enhancing Long Short-Term Memory for Text Categorization
Table 2: Test accuracy, overall skip rate and test time on five benchmark data sets. We apply four different {rt,λ} settings for the different desired skip rate. We implement Skip LSTM and Skim-LSTM using their open-source codes. The results reported in Skim-LSTM on AGNews are (93.60, 30.30, 1.0x). For LSTM-Jump, it uses REINFORCE to train the model and it performs poorly on AGNews. So we do not evaluate it on other data sets.
['Model', '[ITALIC] rt/ [ITALIC] λ', 'AGNews acc', 'AGNews skip', 'AGNews speed', 'DBPedia acc', 'DBPedia skip', 'DBPedia speed', 'Yelp F. acc', 'Yelp F. skip', 'Yelp F. speed', 'Yelp P. acc', 'Yelp P. skip', 'Yelp P. speed', 'Yahoo acc', 'Yahoo skip', 'Yahoo speed']
Compared to LSTM, when skipping about 60% or 90% words, the decline in the accuracy is not significant. Our model even gets better accuracies on AGNews, DBPedia and Yahoo data with a speed up ranging from 1.5x∼1.7x. When the desired skip rate is 0.0 or 0.25, the model improves LSTM across all tasks with an average relative error reduction of 8.0% and 5.7% respectively.