Graph Convolution over Pruned Dependency TreesImproves Relation Extraction
Table 1: Results on TACRED. Underscore marks highest number among single models; bold marks highest among all. † marks results reported in Zhang et al. (2017); ‡ marks results produced with our implementation. ∗ marks statistically significant improvements over PA-LSTM with p
['System', 'P', 'R', 'F1']
We observe that our GCN model outperforms all dependency-based models by at least 1.6 F1. By using contextualized word representations, the C-GCN model further outperforms the strong PA-LSTM model by 1.3 F1, and achieves a new state of the art. In addition, we find our model improves upon other dependency-based models in both precision and recall. Comparing the C-GCN model with the GCN model, we find that the gain mainly comes from improved recall. To understand what the GCN models are capturing and how they differ from a sequence model such as the PA-LSTM, we compared their performance over examples in the TACRED dev set. Specifically, for each model, we trained it for 5 independent runs with different seeds, and for each example we evaluated the model’s accuracy over these 5 runs. For instance, if a model correctly classifies an example for 3 out of 5 times, it achieves an accuracy of 60% on this example. We observe that on 847 (3.7%) dev examples, our C-GCN model achieves an accuracy at least 60% higher than that of the PA-LSTM, while on 629 (2.8%) examples the PA-LSTM achieves 60% higher. We compared the performance of both GCN models with the PA-LSTM on the TACRED dev set. As is shown in the figure, both GCN models capture very different examples from the PA-LSTM model. In the entire dev set of 22,631 examples, 1,450 had at least 3 more GCN models predicting the label correctly compared to the PA-LSTM, and 1,550 saw an improvement from using the PA-LSTM. The C-GCN, on the other hand, outperformed the PA-LSTM by at least 3 models on a total of 847 examples, and lost by a margin of at least 3 on another 629 examples, as reported in the main text. We hypothesize that the diminishing difference results from the LSTM contextualization layer, which incorporates more information readily available at the surface form, rendering the model’s behavior more similar to a sequence model.