Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling
Table 1: Experimental results for different methods on SNLI. |θ|: the number of parameters (excluding word embedding part). Train Accu and Test Accu: the accuracies on training and test sets respectively.
['[BOLD] Model', '| [ITALIC] θ|', '[BOLD] Train Accu', '[BOLD] Test Accu']
For fair comparison, we use 480D Bi-BloSAN, which leads to the similar parameter number with that of baseline encoders. Bi-BloSAN achieves the best test accuracy (similar to DiSAN) among all the sentence encoding models on SNLI. In particular, compared to the RNN models, Bi-BloSAN outperforms Bi-LSTM encoder, Bi-LSTM with attention and deep gated attention by 2.4%, 1.5% and 0.2%, respectively. Bi-BloSAN can even perform better than the semantic tree based models: SPINN-PI encoder (+2.5%)&Tree-based CNN encoder (+3.6%), and the memory network based model: NSE encoder (+1.1%). Additionally, Bi-BloSAN achieves the best performance among the baselines which are based on RNN/CNN/SAN. It outperforms Bi-LSTM (+0.7%), Bi-GRU (+0.8%), Bi-SRU (+0.9%), multi-CNN (+2.5%), Hrchy-CNN (+1.8%) and multi-head attention (+1.5%).