Reinforced Self-Attention Network:a Hybrid of Hard and Soft Attention for Sequence Modeling
Table 1: Experimental results for different methods on SNLI. |θ|: the number of parameters (excluding word embedding part). T(s)/epoch: average training time (second) per epoch. Inference T(s): average inference time (second) for all dev data on SNLI with a batch size of 100.
['[BOLD] Model', '| [ITALIC] θ|', '[BOLD] T(s)/epoch', '[BOLD] Inference T(s)', '[BOLD] Train Accuracy', '[BOLD] Test Accuracy']
Compared to the methods from official leaderboard, ReSAN outperforms all the sentence encoding based methods and achieves the best test accuracy. Specifically, compared to the last best models, i.e., 600D Gumbel TreeLSTM encoders and 600D Residual stacked encoders, ReSAN uses far fewer parameters with better performance. Moreover, in contrast to the RNN/CNN based models with attention or memory module, ReSAN uses attention-only modules with equal or fewer parameters but outperforms them by a large margin, e.g., 600D Bi-LSTM + intra-attention (+3.0%), 300D NSE encoders (+1.7%) and 600D Deep Gated Attn (+0.8%). Furthermore, ReSAN even outperforms the 300D SPINN-PI encoders by 3.1%. , which is a recursive model and uses the result of an external semantic parsing tree as an extra input.