Efficient Contextualized Representation:Language Model Pruning for Sequence Labeling
Table 1: Performance comparison of language models. Models marked with† adopted adaptive softmax and the vanilla LSTMs, which has less softmax parameters. Models marked with♯ employed sampled softmax LSTMs w. projection, which results in less RNN parameters w.r.t. the size of hidden states.
['Network', 'Ind. #', 'Hid. #', 'Layer #', 'Param.# (⋅107) RNN', 'Param.# (⋅107) Others', 'PPL']
The layer-wise dropout ratio is set to 0.5, the RNNs are unrolled for 20 steps without resetting the LSTM states, and the batch size is set to 128. For models without official reported parameter numbers, we estimate their values (marked with†) by assuming they adopted the vanilla LSTM. Note that, for models 3, 5, 6, 7, 8, and 9, PPL refers to the averaged perplexity of the forward and the backward LMs.