IMoJIE: Iterative Memory-Based Joint Open Information Extraction
Table 6: Performance of IMoJIE on aggregated dataset OpenIE-4+ClausIE+RnnOIE, with different filtering techniques. For comparison, SenseOIE trained on multiple system extractions gives an F1 of 17.2 on CaRB.
['[BOLD] Filtering', '[BOLD] Metric Opt. F1', '[BOLD] Metric AUC', '[BOLD] Metric Last F1']
IMoJIE aggregates extractions from multiple systems through the scoring and filtering approach. It uses extractions from OpenIE-4 (190K), ClausIE (202K) and RnnOIE (230K) to generate a set of 215K tuples. We also try two supervised approaches to aggregation, by utilizing the gold extractions from CaRB’s dev set. Extraction Filtering: For every sentence-tuple pair, we use a binary classifier that decides whether or not to consider that extraction. The input features of the classifier are the [CLS]-embeddings generated from BERT after processing the concatenated sentence and extraction. The classifier is trained over tuples from CaRB’s dev set. Sentence Filtering: We use an IMoJIE model (bootstrapped over OpenIE-4), to score all the tuples. Then, a Multilayer Perceptron (MLP) predicts a confidence threshold to perform the filtering. Only extractions with scores greater than this threshold will be considered. The input features of the MLP include the length of sentence, IMoJIE (OpenIE-4) scores, and GPT Radford et al. This MLP is trained over sentences from CaRB’s dev set and the gold optimal confidence threshold calculated by CaRB. We observe that the Extraction, Sentence Filtering are better than no filtering by by 7.5, 11.2 pts in Last F1, but worse at Opt. F1 and AUC. We hypothesise that this is because the training data for the MLP (640 sentences in CaRB’s dev set), is not sufficient and the features given to it are not sufficiently discriminative. Thereby, we see the value of our unsupervised Score-and-Filter that improves the performance of IMoJIE by (3.8, 15.9) pts in (Optimal F1, Last F1). The 1.2 pt decrease in AUC is due to the fact that the IMoJIE (no filtering) produces many low-precision extractions, that inflates the AUC.