Sparse and Continuous Attention Mechanisms
Table 1: Results on IMDB in terms of accuracy (%). For the continuous attentions, we used N∈{32,64,128} Gaussian RBFs N(t,~μ,~σ2), with ~μ linearly spaced in [0,1] and ~σ∈{.1,.5}.
['Attention', '¯ [ITALIC] L≈280']
The accuracies of the individual models are similar, suggesting that continuous attention is as effective as its discrete counterpart, despite having fewer basis functions than words , i.e., N≪L. Among the continuous variants, the sparsemax outperforms the softmax, except for N=64. We also see that a large N is not necessary to obtain good results, which is encouraging for tasks with long sequences. Finally, combining discrete and continuous sparsemax produced the best results, without increasing the number of parameters.