Learning Relational Dependency Networks for Relation Extraction
Table 5: Weak Supervision Results comparing models trained with gold standard examples only (G) and models trained with gold standard and weakly supervised examples combined (G+WS).
['Relation', 'AUC ROC G', 'AUC ROC G+WS', 'F1 G', 'F1 G+W']
To answer Q1, we generated positive training examples using the weak supervision techniques specified earlier. A range of 4 to 8 rules are derived for each relation. Examples for the organization relations countryHQ and foundedBy were generated using standard distant supervision techniques – Freebase databases were mapped to foundedBy while Wikipedia Infoboxes provides entity pairs for countryHQ. Lastly, only 150 weakly supervised examples were utilized in our experiments (all gold standard examples were utilized). Performing larger runs is part of work in progress. We compared our standard pipeline (individually learned relations with only standard features) learned on gold standard examples only versus our system learned with weak and gold examples combined. Surprisingly, weak supervision does not seem to help learn better models for inferring relations in most cases. Only two relations – origin, otherFamily – see substantial improvements in AUC ROC, while F1 shows improvements for age and, otherFamily, and dateFounded. We hypothesize that generating more examples will help (some relations produced thousands of examples), but nonetheless find the lack of improved models from even a modest number of examples a surprising result. Alternatively, the number of gold standard examples provided may be sufficient to learn RDN models. Thus Q1 is answered equivocally, but in the negative.