Mogrifier LSTM
Figure 3: Perplexity vs the rounds r in the PTB ablation study.
['Mogrifier', '54.1']
Fig. Second, we found the low-rank factorization of Qi and Ri to help a bit, but the full-rank variant is close behind which is what we observed on other datasets, as well. Finally, to verify that the alternating gating scheme is not overly complicated, we condition all newly introduced gates on the original inputs x and hprev (see Fig. That is, instead of Eq. the no-zigzag updates are xi =2σ(Qihprev)⊙xi−2 for odd i∈[1…r], hiprev =2σ(Rix)⊙hi−2prev for even i∈[1…r]. In our experiments, the no-zigzag variant underperformed the baseline Mogrifier by a small but significant margin, and was on par with the r=2 model in Fig.