Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction
Table 3: Performance comparison on the two datasets.
['Model', 'NYT29 Prec.', 'NYT29 Rec.', 'NYT29 F1', 'NYT24 Prec.', 'NYT24 Rec.', 'NYT24 F1']
Among the baselines, HRL achieves significantly higher F1 scores on the two datasets. Our WordDecoding (WDec) model achieves F1 scores that are 3.9% and 4.1% higher than HRL on the NYT29 and NYT24 datasets respectively. Similarly, our PtrNetDecoding (PNDec) model achieves F1 scores that are 3.0% and 1.3% higher than HRL on the NYT29 and NYT24 datasets respectively. We perform a statistical significance test (t-test) under a bootstrap pairing between HRL and our models and see that the higher F1 scores achieved by our models are statistically significant (p<0.001). Next, we combine the outputs of five runs of our models and five runs of HRL to build ensemble models. For a test instance, we include those tuples which are extracted in the majority (≥3) of the five runs. This ensemble mechanism increases the precision significantly on both datasets with a small improvement in recall as well. In the ensemble scenario, compared to HRL, WDec achieves 4.2% and 3.5% higher F1 scores and PNDec achieves 4.2% and 2.9% higher F1 scores on the NYT29 and NYT24 datasets respectively. Both datasets have a similar set of relations and similar texts (NYT). So task-wise both datasets should pose a similar challenge. However, the F1 scores suggest that the NYT24 dataset is easier than NYT29. The reason is that NYT24 has around 72.0% of overlapping tuples between the training and test data (% of test tuples that appear in the training data with different source sentences). In contrast, NYT29 has only 41.7% of overlapping tuples. Due to the memorization power of deep neural networks, it can achieve much higher F1 score on NYT24. The difference between the F1 scores of WordDecoding and PtrNetDecoding on NYT24 is marginally higher than NYT29, since WordDecoding has more trainable parameters (about 27 million) than PtrNetDecoding (about 24.5 million) and NYT24 has very high tuple overlap. However, their ensemble versions achieve closer F1 scores on both datasets.