Not All Attention Is Needed: Gated Attention Network for Sequence Data
Table 5: Experiment results on IMDB Reviews with different auxiliary networks in GA-Net.
['GA-Net', 'Accuracy', 'Density']
Both BiLSTM+AUXFNN and BiLSTM+AUXATT outperform all baseline models. This proves again that not all attention is needed, and the ability of auxiliary network in selecting meaningful units. Compared with BiLSTM+AUXFNN and BiLSTM+AUXATT, BiLSTM+AUXLSTM is still the one who achieves the best performance considering both accuracy and density. Therefore, LSTM is a relatively good choice for auxiliary network in dealing with similar tasks in which the inputs are sequences.