DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
(a) LSTM-based language model (ours) on WT103. ⋆ For this experiment, we use two GPUs.
['[BOLD] Row', '[BOLD] Configuration  [BOLD] Input-Output', '[BOLD] Configuration  [BOLD] Depth of', '[BOLD] Configuration  [BOLD] Dimension of', '[BOLD] Parameter Distribution (in millions)  [BOLD] DeFINE', '[BOLD] Parameter Distribution (in millions)  [BOLD] Context', '[BOLD] Parameter Distribution (in millions)  [BOLD] Input-Output', '[BOLD] Parameter Distribution (in millions)  [BOLD] Total', '[BOLD] Training  [BOLD] Time', '[BOLD] Perplexity  [BOLD] Val', '[BOLD] Perplexity  [BOLD] Test']
Though the adaptive input (baevski2018adaptive) and output ( grave2017efficientSoft) methods are effective and reduce the number of parameters significantly, our method further improves performance by about 3 points while learning only 1.25% (or 0.4 million) more parameters.