Integrating Local Context and Global Cohesivenessfor Open Information Extraction
Table 4. Performance comparison with state-of-the-art Open IE systems on two datasets from different domains, using Precision@K, Mean Average Precision (MAP), Normalized Discounted Cumulative Gain (NDCG) and Mean Reciprocal Rank (MRR).
['[BOLD] Methods', '[BOLD] NYT ( riedel2013relation,  ) P@100', '[BOLD] NYT ( riedel2013relation,  ) P@200', '[BOLD] NYT ( riedel2013relation,  ) MAP', '[BOLD] NYT ( riedel2013relation,  ) NDCG@100', '[BOLD] NYT ( riedel2013relation,  ) NDCG@200', '[BOLD] NYT ( riedel2013relation,  ) MRR', '[BOLD] Twitter ( zhang2016geoburst,  ) P@100', '[BOLD] Twitter ( zhang2016geoburst,  ) P@200', '[BOLD] Twitter ( zhang2016geoburst,  ) MAP', '[BOLD] Twitter ( zhang2016geoburst,  ) NDCG@100', '[BOLD] Twitter ( zhang2016geoburst,  ) NDCG@200', '[BOLD] Twitter ( zhang2016geoburst,  ) MRR']
2. Performance on Relation Tuple Extraction. On NYT and twitter test set, we compare ReMine with its variants ReMine-L and ReMine-G as well as four baseline open IE systems mentioned above. All methods experience performance drop in Twitter, while ReMine declines less than any other methods on the rank-based measures. In the NYT dataset, all the systems except OLLIE have similar overall precision (i.e. P@300). But ReMine has a “higher” curve since most tuples obtained by Stanford OpenIE and ClausIE will be assigned score 1. Therefore we may not rank them in a very rational way. In contrast, the scores of different tuples obtained by ReMine-G and ReMine are usually distinct from each other. In the Twitter dataset, ReMine shows its power in dealing with short and noisy text. Both ClausIE and MinIE have a rather low score since there are lots of non-standard language usages and grammatical errors in tweets. In twitter, dependency parsing attaches more wrong arguments and labels than usual. All methods investigated depend on dependency parsing to varying degrees, while clause-based methods rely heavily on it and may not achieve a satisfying performance. Model-wise, we believe global cohesiveness helps open IE from two aspects: (1) ranking tuples (2) updating entity phrase pairs. In particular, ReMine-G differs from ReMine-L only on extraction scores, since global cohesiveness σ provides better ranking performance (P@300) over random (ReMine-L). The gain between ReMine and ReMine-G clearly shows the updated extractions have better quality in general.