Contextualised Graph Attention for Improved Relation Extraction
Table 2: Performance of various models on SemEval test set. P: Precision; R: Recall; F: F1-score
['Model', 'P', 'R', 'F']
The c+gat+mg+dref model using dependency relation-based edge features achieves the best F1-score of 86.30. The c+gat+mg+dref model scores both in terms of higher precision (86.03) and recall (86.66) to achieve a higher F1-score, indicating the usefulness of the proposed model. The GCN models using dref and ctef edge features also report comparatively higher F1-scores of 85.82 and 85.91, respectively. The higher performance of c+gat+dref model mainly due to its higher recall (87.56). However, c+gat+ctef scores a higher precision (87.24) using ctef features. For example, the performance of c+gat+mg model using both vertex features and edge features (ctef and dref) is higher than that of c+gat+mg, which considers only vertex features. A similar result is observed across other multiple graph-based models such as gat+mg, c+gcn+mg and gcn+mg. Although a higher performance is achieved using dref and ctef edge features individually, combining dref and ctef does not help in further improving the performance. We see that using a contextual layer to provide vertex features for GCN and GAT layers is useful for obtaining better performance. The performance of GAT-based models is comparatively higher than GCN-based models for relation extraction. The ability of GATs to attend to neighbouring vertices and edge features when computing vertex representations is more useful than simply considering the structural information as in the case of GCNs to achieve higher performance. The performance of gcn, c+gcn, gat and c+gat models using multiple graphs scores significantly higher than its counterparts using only a single graph. The improvement is not limited to models that use edge features but holds true for models that do not use edge features as well. For example, results of c+gat+mg (F1-score of 85.12), which uses multiple sub-graphs (without edge features) is higher than c+gat+sg (F1-score of 83.52), which uses a single graph. The same results can be seen across other models: gat+mg (84.80) vs. gat+sg (80.80); c+gcn+mg (84.83) vs. c+gcn+sg (83.84) and gcn+mg (84.40) vs. gcn+sg (80.50). The above results sufficiently establish that using segregated smaller sub-graphs as opposed to a single graph is more useful in learning richer vertex representations for relation extraction.