Adaptive Input Representations for Neural Language Modeling
Table 8: Validation perplexity on wikitext-103 with tied adaptive inputs & outputs. The bands signify the number of words belonging to each band. In every case, the first band has dimension 1024, the second band 256, the third band 64 and the fourth band (if it exists) 16.
['[BOLD] Softmax cutoff', '[BOLD] Valid PPL']
This appendix shows various ablation. We also experimented with sharing the head projection but found this to perform less well than not sharing it.