Learning from Explanations with Neural Execution Tree
Table 2: Experiment results on Relation Extraction and Sentiment Analysis. Average and standard deviation of F1 scores (%) over multiple runs are reported (5 runs for RE and 10 runs for SA). LF(E) denotes directly applying logical forms onto explanations. Bracket behind each method illustrates corresponding data used in the method. S denotes training data without labels, E denotes explanations, R denotes surface pattern rules transformed from explanations; Sa denotes labeled data annotated with explanations, Su denotes the remaining unlabeled data. Sl denotes labeled data annotated using same time as creating explanations E, Slu denotes remaining unlabeled data corresponding to Sl.
['[EMPTY]', 'TACRED', 'SemEval']
We observe that our proposed NExT consistently outperforms all baseline models in low-resource setting. Also, we found that, (1) directly applying logical forms to unlabeled data results in poor performance. We notice that this method achieves high precision but low recall. Based on our observation of the collected dataset, this is because people tend to use detailed and specific constraints in an NL explanation to ensure they cover all aspects of the instance. As a result, those instances that satisfy the constraints are correctly labeled in most cases, and thus the precision is high. Meanwhile, generalization ability is compromised, and thus the recall is low. (2) Compared to its downstream classifier baseline (BiLSTM+ATT with Sa), NExT achieves 4.2% F1 improvement in absolute value on TACRED, and 5.5% on SemEval. This validates that the expansion of rule coverage by NExT is effective and is providing useful information to classifier training. (3) Performance gap further widens when we take annotation efforts into account. The annotation time for E and Sl are equivalent; but the performance of BiLSTM+ATT significantly degrades with fewer instances in Sl. (4) Results of semi-supervised methods are unsatisfactory. This may be explained with difference between underlying data distribution of Sa and Su.