Multi-Zone Unit for Recurrent Neural Networks
Table 4: Ablation study of M-functions in MZUs. “Regular” stands for replacing the M-function with the regular linear transformation function on [xt,ht−1].
['Model', 'No-DT', 'DT']
Since the M-function works on the gating module (i.e., gt) and the transformation module (i.e., ˜ht), we conduct an ablation study to investigate its effects on both modules. MZU models, since replacing any module with the Regular one sharply declines the performance. Even with the module degeneration, our MZUs still outperform GRUs which only based on single space composition, in shallow and deep models, correspondingly.