Not All Attention Is Needed: Gated Attention Network for Sequence Data
Table 2: Results of classification accuracy on various datasets for text classification. Bold numbers indicate best performance. Numbers in brackets indicate density of attention connections.
['[EMPTY]', 'IMDB', 'AG’s News', 'SST-1', 'SST-2', 'TREC']
As it can be seen that, the proposed GA-Net consistently achieves the best performance on all datasets. For example, in TREC dataset, GA-Net achieves accuracy of 91.24%, and outperforms the baselines by at least 1.16%. At the same time, for the density of the resulting attention connections, we can see that it achieves this with much sparser attention structures. Especially for long sequences as in IMDB dataset where the average sequence length is 231, only 19.99% gates are switched on for each input. It demonstrates the fact that not all attention is needed especially for long sequences, and that GA-Net indeed has the ability of selecting those important units to attend to. In BiLSTM+localAtt, we choose a window size of 40, 16, 8, 8, 4 for each dataset respectively. This aim at making BiLSTM+localAtt have similar sparsity with GA-Net and achieve good performance at the same time.