Tensorized Embedding Layers
Table 2: NMT, Transformer-big on WMT‘14 English-to-German dataset. Both case-sensitive tokenized BLEU (higher is better) and de-tokenized SacreBLEU (post2018) on newstest2014 are reported. In case of low-rank (LR) factorization, rank is the factorization rank; in case of TT-embedding (TT), rank is the TT-rank.
['[BOLD] Model', '[BOLD] Embedding shape', '[BOLD] Rank', '[BOLD] Token', '[BOLD] Sacre', '[BOLD] Emb', '[BOLD] Total']
We observe that even in this rather challenging task, both embedding and softmax layers can be compressed significantly, at the cost of a small drop in the BLEU score. However, with the increase of compression factor, the performance deteriorates rapidly. Compared to the sentiment analysis, NMT is a much more complex task which benefits more from additional capacity (in the form of more powerful RNN or more transformer blocks) rather than regularization (bahdanau2014neural; vaswani2017attention; wu2019pay), which may explain why we did not manage to improve the model by regularizing its embedding layers with TT-embedding.