TensorCoder: Dimension-Wise Attention via Tensor Representation for Natural Language Modeling
Table 1: Valid Loss and FLOPs on PTB and Wiki-103.
['[BOLD] Model', '[BOLD] PTB valid Loss', '[BOLD] PTB Attention GFLOPs', '[BOLD] Wiki-103 valid Loss', '[BOLD] Wiki-103 Attention GFLOPs']
We use short sequences (the length is 100) on the smaller PTB dataset, and use long sequences (the length is 160) on the larger wiki-103 dataset. In PTB dataset, the two convolution methods of TensorCoder have lower loss compared with the Transformer, and the model complexity is lower, the maximum is 3.05 times. In Wiki-103 dataset, the loss and FLOPs of TensorCoder are much smaller than Tensorized Transformer. More complexity comparison and hyperparameter settings can be found in Supplementary Materials IV model complexity comparison.