NERO: A Neural Rule Grounding Framework forLabel-Efficient Relation Extraction
Table 3. Performance on predicting unseen relations. Nero applies the learned soft rule matcher on unseen relation rules to make predictions.
['Method', 'TACRED P', 'TACRED R', 'TACRED  [ITALIC] F1', 'SemEval P', 'SemEval R', 'SemEval  [ITALIC] F1']
3. Performance on Unseen Relations. To show that Nero has the capacity of predicting unseen relations, we evaluate Nero on relations unseen in training. In the experiment, we randomly sample 5 relations as unseen relations and repeat the experiment 10 times. In each time, we use the same sets of unseen relations and the same random seed for all methods. We remove the sampled relations (both sentences and rules) in training while keeping only them in testing. We sample sentences of other relations as None relation in testing with the same ratio as the raw corpus (79.5% for TACRED and 17.4% for SemEval). Moreover, since the special tokens subj/obj-ner are out-of-vocabulary in both Glove and BERT, we only use the words between two entities in encoding and only use rules with the same entity types to the sentence in inference. To make predictions by Nero, we find the most similar rule using soft rule matcher. It shows that instead of just memorizing the rules and the corresponding relation types, the soft rule matcher also learns knowledge about how to better perform rule matching and can even generalize to unseen relations.