TensorCoder: Dimension-Wise Attention via Tensor Representation for Natural Language Modeling
Table 2: Bleu and FLOps in WMT16
['[BOLD] Model', '[BOLD] BLEU', '[BOLD] FLOPs']
In neural machine translation, we have replaced the attention with dimension-wise attention in each layer, with other parts remaining the same. The length of sequence is set to the maximum length of the sentence in the training set. ’ ∗’ indicates that the result is our own implementation, ’−’ indicates no reported results in that setting. In WMT16 dataset, the bleu score of TensorCoder (1conv and 8conv) are 30.7 and 33.1, which are similar performance over Transformer. However, TensorCoder has lower FLOPs, the FLOPs of Transformer is 3.5 times over our model under similar performance. In IWSLT16, The TensorCoder aslo has lower FLOPs, the FLOPs of TensorCoder only is 1/3 of Transformer.