Higher-order Relation Schema Induction using Tensor Factorization with Back-off and Aggregation
Table 5: Higher-order RSI accuracies of various methods on the three datasets. Induced schemata for each dataset and method are evaluated by three human evaluators, E1, E2, and E3. TFBA performs better than HardClust for Shootings and NYT Sports datasets. Even though HardClust achieves better accuracy on MUC dataset, it has several limitations, see Section 4 for more details. Chambers-13 solves a slightly different problem called event schema induction, for more details about the comparison with Chambers-13 see Section 4.1.
['[EMPTY]', 'Shootings E1', 'Shootings E2', 'Shootings E3', 'Shootings Avg', 'NYT Sports E1', 'NYT Sports E2', 'NYT Sports E3', 'NYT Sports Avg', 'MUC E1', 'MUC E2', 'MUC E3', 'MUC Avg']
We present evaluation results from three evaluators represented as E1, E2 and E3. Percentage agreement of the evaluators for TFBA is 72%, 70% and 60% for Shootings, NYT Sports and MUC datasets respectively. HardClust also performs better than Chambers-13 on all the datasets. From this analysis we infer that there is a need for algorithms which induce higher-order schemata for relations, a gap we fill in this paper. Example schemata induced by TFBA and (Chambers-13) are provided as part of the supplementary material.