Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling
Table 3: An ablation study of Bi-BloSAN. “Local” denotes the local context representations h and “Global” denotes the global context representations E. “Bi-BloSAN w/o mBloSA” equals to word embeddings directly followed by a source2token attention and “Bi-BloSAN w/o mBloSA & source2token self-attn.” equals to word embeddings plus a vanilla attention without q.
['[BOLD] Model', '| [ITALIC] θ|', '[BOLD] Test Accuracy']
In particular, we evaluate the contribution of each part of Bi-BloSAN by the change of test accuracy after removing the part. The removed part could be: 1) local context representations h, 2) global context representations E, 3) the context fusion module (mBloSA) or 4) all fundamental modules appeared in this paper. The results show that both the local and global context representations play significant roles in Bi-BloSAN. They make Bi-BloSAN surpass the state-of-the-art models. Moreover, mBloSA improves the test accuracy from 83.1% to 85.7%. Source2token self-attention performs much better than vanilla attention, and improves the test accuracy by 3.3%.