DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
Table 1: Experimental results for different methods on SNLI. |θ|: the number of parameters (excluding word embedding part). T(s)/epoch: average time (second) per epoch. Train Accu(%) and Test Accu(%): the accuracy on training and test set.
['[BOLD] Model Name', '| [ITALIC] θ|', '[BOLD] T(s)/epoch', '[BOLD] Train Accu(%)', '[BOLD] Test Accu(%)']
DiSAN surpasses the RNN/CNN based models with more complicated architecture and more parameters by large margins, e.g., +2.32% to Bi-LSTM, +1.42% to Bi-LSTM with additive attention. It even outperforms models with the assistance of a semantic parsing tree, e.g., +3.52% to Tree-based CNN, +2.42% to SPINN-PI. First, a comparison between the first two models shows that changing token-wise attention to multi-dimensional/feature-wise attention leads to 3.31% improvement on a word embedding based model. Also, a comparison between the third baseline and DiSAN shows that DiSAN can substantially outperform multi-head attention by 1.45%. Moreover, a comparison between the forth baseline and DiSAN shows that the DiSA block can even outperform Bi-LSTM layer in context encoding, improving test accuracy by 0.64%. A comparison between the fifth baseline and DiSAN shows that directional self-attention with forward and backward masks (with temporal order encoded) can bring 0.96% improvement. It is ×3 faster than widely used Bi-LSTM model. Compared to other models with competitive performance, e.g., 600D Bi-LSTM encoders with intra-attention (2.8M), 300D NSE encoders (3.0M) and 600D Bi-LSTM encoders with multi-dimensional attention (2.88M), DiSAN only has 2.35M parameters.