Indirect Supervision for Relation Extraction using Question-Answer Pairs
Table 6. Performance comparison on end-to-end relation extraction (at the highest F1 point) on the two datasets.
['[BOLD] Method', '[BOLD] NYT\xa0(Riedel et\xa0al.,  2010 ; Hoffmann et\xa0al.,  2011 )  [BOLD] Prec', '[BOLD] NYT\xa0(Riedel et\xa0al.,  2010 ; Hoffmann et\xa0al.,  2011 )  [BOLD] Rec', '[BOLD] NYT\xa0(Riedel et\xa0al.,  2010 ; Hoffmann et\xa0al.,  2011 )  [BOLD] F1', '[BOLD] NYT\xa0(Riedel et\xa0al.,  2010 ; Hoffmann et\xa0al.,  2011 )  [BOLD] Time', '[BOLD] KBP\xa0(Ellis et\xa0al.,  2014 ; Ling and Weld,  2012 )  [BOLD] Prec', '[BOLD] KBP\xa0(Ellis et\xa0al.,  2014 ; Ling and Weld,  2012 )  [BOLD] Rec', '[BOLD] KBP\xa0(Ellis et\xa0al.,  2014 ; Ling and Weld,  2012 )  [BOLD] F1', '[BOLD] KBP\xa0(Ellis et\xa0al.,  2014 ; Ling and Weld,  2012 )  [BOLD] Time']
Performance Comparison with Baselines. To test the effectiveness of our proposed framework ReQuest, we compare with other methods on the relation extraction task. The precision, recall, F1 scores as well as the model As shown in the table, ReQuest achieves superior F1 score on both datasets compared with other models. Among all these baselines, MultiR and CoType-RM handle noisy training data while the remaining ones assume the training corpus is perfectly labeled. Due to their nature of being cautious towards the noisy training data, both MultiR and CoType-RM reach relatively high results confronting with other models that blindly exploit all heuristically obtained training examples. However, as external reliable information sources are absent and only the noise from multi-label relation mentions (while none or only one assigned label is correct) is tackled in these models, MultiR and CoType-RM underperform ReQuest. Especially from the comparison with CoType-RM, which is also an embedding learning based relation extraction model with the idea of partial-label loss incorporated, we can conclude that the extra semantic inklings provided by the QA corpus do help boost the performance of relation extraction. Performance Comparison with Ablations. We experiment with two variations of ReQuest, ReQuest-QA_RE and ReQuest-RE_QA, in order to validate the idea of joint optimization. However, their results still fall behind ReQuest’s. Thus, separately training the two components may not capture as much information as jointly optimizing the combined objective. The idea of constraining each component in the joint optimization process proves to be effective in learning embeddings to present semantic meanings of objects (e.g. features, types and mentions).