1 Introduction
Table 2: Test-set negative log likelihoods while holding number of parameters approximately constant. Subword modeling tasks on Penn Treebank and IMDB.
['[BOLD] PTB-SW', '[BOLD] Performance', '[EMPTY]']
This simple partitioning scheme was designed to ensure that no subword was shorter than two characters in length. Future work will entail designing a more realistic subword partitioning algorithm. Subwords below a certain frequency were discarded, and combined with 26 single characters to create the final dictionary. For Penn Treebank, this yields a vocabulary of 2405 symbols was created (2,378 subwords + 26 characters + 1 end-token). For the IMDB corpus, after replacing all emoticons and special non-word symbols with special tokens, we obtain a dictionary of 1926 symbols (1899 subwords + 26 single characters + 1 end-token). With respect to the word and character-level benchmarks, we see that the Delta-RNN outperforms all previous, un-regularized models, and performs comparably to regularized state-of-the-art. This extends the results of Wu et al. However, the Delta-RNN, which also makes use of second-order units within its inner function, ultimately offers the best performance and performs better than the LSTM in all experiments. In both Penn Treebank and IMDB subword language modeling experiments, the Delta-RNN is competitive with complex architectures such as the GRU and the MGU. In both cases, the Delta-RNN nearly reaches the same performance as the best performing baseline model in either data-set (i.e., it nearly reaches the same performance as the GRU on Penn Treebank and the MGU on IMDB). Surprisingly, on IMDB, a simple Elman network is quite performant, even outperforming the MI-RNN. We argue that this might be the result of constraining all neural architectures to only a small number of parameters for such a large data-set, a constraint we intend to relax in future work.