MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
Table 3: Comparisons between MUSE and its variants on the IWSLT 2015 De-En translation task.
['Model', 'BLEU']
Does concatenating self-attention with convolution certainly improve the model? To bridge the gap between point-wise transformation which learns token level representations and self-attention which learns representations of global context, we introduce convolution to enhance our multi-scale attention. However, it is not easy to combine convolution and self-attention in one module to build better representations on sequence to sequence tasks. Furthermore, combining depth-wise separable convolution (in this work we choose its best variant dynamic convolution as implementation) is even worse than combining convolution. Why do we choose DepthConv and what is the importance of sharing Projection of DepthConv and self-attention? We conjecture that convolution and self-attention both learn contextual sequence representations and they should share the point transformation and perform the contextual transformation in the same hidden space. We first project the input to a hidden representation and perform a variant of depth-wise convolution and self-attention transformations in parallel. There are several studies on combining self-attention and convolution. However, they do not surpass both convectional and self-attention mechanisms. Sukhbaatar et al. Since state-of-the-art models on question answering tasks still consist on self-attention and do no adopt ideas in QAnet It seems that learning global and local context through stacking self-attention and convolution layers does not beat either self-attention or convolution models. In contrast, the proposed parallel multi-scale attention outperforms previous convolution or self-attention based models on main translation tasks, showing its effectiveness for sequence to sequence learning.