An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
Table 3: LSTM parameter settings for experiments in Section 5.
['LSTM Settings (Key Parameters)  [BOLD] Dataset/Task', 'LSTM Settings (Key Parameters)  [BOLD] Subtask', 'LSTM Settings (Key Parameters)  [ITALIC] n', 'LSTM Settings (Key Parameters)  [BOLD] Hidden', 'LSTM Settings (Key Parameters)  [BOLD] Dropout', 'LSTM Settings (Key Parameters)  [BOLD] Grad Clip', 'LSTM Settings (Key Parameters)  [BOLD] Bias', 'LSTM Settings (Key Parameters)  [BOLD] Note']
These values are picked from hyperparameter search for LSTMs that have up to 3 layers, and the optimizers are chosen from {SGD, Adam, RMSprop, Adagrad}. For certain larger datasets, we adopted the settings used in prior work (e.g., Grave et al. GRU hyperparameters were chosen in a similar fashion, but typically with more hidden units than in LSTM to keep the total network size approximately the same (since a GRU cell is more compact).