SesameBERT: Attention for Anywhere
Table 3: Comparison of specified layers among various approaches in the RTE dataset. We dissected our models into two methods. SE-BERT refers to BERT with Squeeze and Excitation; BLUR-BERT refers to BERT with Gaussian blurring.
['[BOLD] Layers', '[BOLD] BERT', '[BOLD] SE-BERT', '[BOLD] BLUR-BERT', '[BOLD] SesameBERT']
Gong2019 demonstrated that different layers vary in terms of their abilities to distinguish and capture neighboring positions and global dependency between words. We evaluated the weights learned from all layers. These weights indicated that a heavier weight represents greater importance. Because the lower layer represents word embeddings that are deficient in terms of context (Yang2018), the self-attention model in the lower layer may need to encode representations with global context and may struggle to learn localness. The results indicated that the lower layers had lower accuracy.