Effective Attention Modeling for Neural Relation Extraction
Table 4: Effectiveness of model components (m=4) on the NYT11 dataset.
['[EMPTY]', 'Prec.', 'Rec.', 'F1']
We include the ablation results on the NYT11 dataset in Table 4. When we add multi-factor attention to the baseline BiLSTM-CNN model without the dependency distance-based weight factor in the attention mechanism, we get 0.8% F1 score improvement (A2−A1). Adding the dependency weight factor with a window size of 5 improves [CONTINUE] the F1 score by 3.2% (A3−A2). Increasing the window size to 10 reduces the F1 score marginally (A3−A4). Replacing the attention normalizing function with softmax operation also reduces the F1 score marginally (A3−A5). In our model, we concatenate the features extracted by each attention layer. Rather than concatenating them, we can apply max-pooling operation across the multiple attention scores to compute the final attention scores. These max-pooled attention scores are used to obtain the weighted average vector of Bi-LSTM hidden vectors. This affects the model performance negatively and F1 score of the model decreases by 3.0% (A3−A6).