Mogrifier LSTM
Table 4: Hyperparameter tuning ranges for all tasks except Enwik8.
['[EMPTY]', 'Low', 'High', 'Spacing']
In all experiments, we tuned hyperparameters using Google Vizier ( Golovin et al. Obviously, mogrifier_rounds and mogrifier_rank are tuned only for the Mogrifier. If input_embedding_ratio⩾1, then the input/output embedding sizes and the hidden sizes are set to equal and the linear projection from the cell output into the output embeddings space is omitted. Similarly, mogrifier_rank⩽0 is taken to mean full rank Q∗, R∗ without factorization. Since Enwik8 is a much larger dataset, we don’t tune input_embedding_ratio and specify tighter tuning ranges for dropout based on preliminary experiments