Reinforced Self-Attention Network:a Hybrid of Hard and Soft Attention for Sequence Modeling
Table 2: An ablation study of ReSAN.
['[BOLD] Model', '| [ITALIC] θ|', '[BOLD] Inference T(s)', '[BOLD] Test Accu.']
One by one, each component is removed and the changes in test accuracy are recorded. the hard attention module, 2) soft self-attention module and 3) both hard attention and soft self-attention modules. In terms of prediction quality, the results show that 1) the unselected head tokens do contribute to the prediction, bringing 0.2% improvement; 2) using separate RSS modules to select the head and dependent tokens improves accuracy by 0.5%; and 3) hard attention and soft self-attention modules improve the accuracy by 0.3% and 2.9% respectively. In terms of inference time, it shows that 1) the two variants are more time-efficient but have poorer performance; and 2) applying the RSS modules to self-attention or attention improves not only performance but also time efficiency.