Gated Group Self-Attention for Answer Selection
Table 4. Effectiveness of offset, global information gate and interaction mechanism. All models use max-pooling for composition.
['Model (insuranceQA)', 'Test1', 'Test2']
Effectiveness of Offset Strategy, Global Information Gate and Interaction Mechanism We also prove the effectiveness of offset strategy, global information gate and interaction mechanism. Group self-attention is the model in which the global multi-head attention is simply replaced by group multi-head attention without offset and global information gate. We can see that the accuracy of group self-attention is lower than that of global self-attention. This is reasonable because group self-attention itself does not have the ability to model global information. By adding offset strategy to group self-attention, the accuracy is improved. This improvement is due to the expansion of its receptive field, which proves the effectiveness of offset strategy. The accuracy can be further improved by adding global information gate, which constitutes our proposed model GGSA finally. GGSA outperforms global self-attention, which indicates that it is effective to distinguish local information from global information. This result also indicates that the global information gate can incorporate the global information into the representation of words, which proves that the motivation of global information gate is right. Furthermore, iGGSA gains an improvement of 3% compared to GGSA, which proves the effectiveness of interaction mechanism proposed in this paper.