DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
(a) Different transformations (see Figure 2)
['[BOLD] Layer', '[BOLD] # Parameters  [BOLD] (in millions)', '[BOLD] Perplexity  [BOLD] Val', '[BOLD] Perplexity  [BOLD] Test']
HGT is as effective as linear transformation while learning two million fewer parameters. Compared to group linear transform (GLT), HGT improves perplexity by about 5 points while learning a similar number of parameters.