Multi-Zone Unit for Recurrent Neural Networks
Table 2: BPC on text8 test set.
['Model', 'Size', 'BPC']
To demonstrate that MZUs work well across datasets, we conduct experiments on a larger dataset text8. Explicit using longer context indeed improves the performance of language modeling. While in this work, we mainly focus on boosting the transition functions (i.e., only operate on xt and ht−1) of RNNs. Our approaches can also be applied on transformer-based architectures, e.g for enhancing the multi-head attention.