Multi-Zone Unit for Recurrent Neural Networks
Table 3: Effects of zone numbers on Penn Treebank test set. All the tested models are with “DT”, which omitted due to the limited table space.
['#Zones', '[ITALIC] satMZU', '[ITALIC] gcnMZU', '[ITALIC] capMZU']
-Sx3SSx1SSSx3Px4 The Number of Zones. We investigate the impact of zone numbers on MZUs, in particular, we conduct experiments with three deep MZU models (MZU+DT) on Penn Treebank. For each model, we respectively test 2, 4, 8, and 16 zones in M-functions. As the dimension of the zone is decided by the hidden size and the zone number, i.e., dz = dh/N, more zones do not lead to more parameters. Additionally, we observe that the satMZU is more sensitive on the number of zones, and leveraging too many zones (e.g. 8, 16) sharply declines the performance. The gcnMZU and the capMZU are more robust on modeling multiple zones, benefiting from the powerful graph convolutional structure and dynamic routing algorithm, respectively.