Generating Wikipedia by Summarizing Long Sequences
Table 4: Performance of best models of each model architecture using the combined corpus and tf-idf extractor.
['[BOLD] Model', '[BOLD] Test perplexity', '[BOLD] ROUGE-L']
Abstractive model architecture and input length: This motivated the Transformer-Decoder, which we found could learn and improve up to L=4000, before running out of memory on our machines equipped with 16GB of GPU RAM (NVIDIA P100). By using the T-DMCA modifications, we were able to train up to L=11000 and continued to see improvements in performance. We also found the MoE-layer helped performance by adding model capacity at high L, for example dropping log-perplexity from 2.05 to 1.93 at L=11000 with 128 experts. Our best model attempted uses 256 experts at L=7500 (we were unable to use 256 experts with L=11000 due to memory constraints) and achieves a perplexity of 1.90,