Accessing Higher-level Representations in Sequential Transformers with Feedback Memory
Table 4: Results on WikiText-103. We compare with the state of the art on word level language modeling. We report perplexity (ppl) for the dev and test sets as well as the number of parameters.
['Model', 'Params', 'Dev', 'Test']
Our small model outperformed some of the Transformer baselines with more than 100M parameters despite having only 40M parameters. Our large model matched the performance of the TransformerXL with almost half the parameters. Being able to maintain strong performance while reducing size is important for applications where memory is a concern.