DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
(a) Depth (N) vs width (k)
['[BOLD] Depth of  [BOLD] DeFINE\xa0( [ITALIC] N)', '[BOLD] Dimensions of  [BOLD] e [ITALIC] i ( [ITALIC] n)', '[BOLD] Dimensions of  [BOLD] e [ITALIC] o ( [ITALIC] m)', '[BOLD] Dimensions of  [BOLD] ^e [ITALIC] i ( [ITALIC] k)', '#  [BOLD] Parameters  [BOLD] (in millions)', '[BOLD] Perplexity  [BOLD] Val', '[BOLD] Perplexity  [BOLD] Test']
For the same value of k, the performance of the language model improves with the increase in the depth N. However, when we scale the width k for a fixed value of depth N, the performance does not improve. This is likely because, as we increase the size of k, more neurons are receiving their input from the same subset of dimensions and thus learning many redundant parameters.