An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
Table 1: Evaluation of TCNs and recurrent architectures on synthetic stress tests, polyphonic music modeling, character-level language modeling, and word-level language modeling. The generic TCN architecture outperforms canonical recurrent networks across a comprehensive suite of tasks and datasets. Current state-of-the-art results are listed in the supplement. h means that higher is better. ℓ means that lower is better.
['Sequence Modeling Task', 'Model Size (≈)', 'Models Model Size (≈)', 'Models LSTM', 'Models GRU', 'Models RNN', '[BOLD] TCN']
Note that on several of these tasks, the generic, canonical recurrent architectures we study (e.g., LSTM, GRU) are not the state-of-the-art. (See the supplement for more details.) With this caveat, the results strongly suggest that the generic TCN architecture with minimal tuning outperforms canonical recurrent architectures across a broad variety of sequence modeling tasks that are commonly used to benchmark the performance of recurrent architectures themselves. We now analyze these results in more detail. We now discuss the results on polyphonic music modeling, character-level language modeling, and word-level language modeling. We mention some of these specialized architectures when useful, but our primary goal is to compare the generic TCN model to similarly generic recurrent architectures, before domain-specific tuning.