,
Table 19: Generalization to unseen terms: Effectiveness of various methods, using the same metrics as in Table 18. As expected, supervised methods perform poorly, similar to a random baseline. Unsupervised methods generalize well, but the ET classifier again substantially outperforms them.
['[EMPTY]', 'AUC', 'Acc', 'P', 'R', 'F1']
As a second study, we assess how well our classifier generalizes to unseen terms. For this, we consider only the 559 test terms that do not appear in the train set. We see that the frequency based supervised baselines, having never seen the test terms, stay close to the default precision of 0.5. The unsupervised baselines, by nature, generalize much better but are substantially dominated by our ET classifier, which achieves an F1 score of 78%. This is only 2% below its own F1 across all seen and unseen terms, and 6% higher than the second best baseline.