Multi-Zone Unit for Recurrent Neural Networks
Table 1: BPC on Penn Treebank test set.
['Model', 'Size', 'BPC']
Our MZU models outperform previous competitive approaches, including deeper RNNs, e.g., the 4-layers FS-LSTM and the 11-layers res-IndRNN, without using more parameters. Among three MZU models, the gcnMZU+DT and the capMZU+DT achieve better performance than the satMZU+DT, and the capMZU+DT achieves the best BPC with 1.181. The capMZU can model part-whole relationships between the old zones and the new zones to aggregate useful information, through multiple iterations of routing-by-agreement. Such kind of multi-round refinements let the new zones contain more accurate representations and therefore capMZU preforms better than other MZUs.