Imposing Label-Relational Inductive Biasfor Extremely Fine-Grained Entity Typing
Table 2: Comparison with baseline models on the Ultra-Fine dataset. Threshold-tuning gives better performance on all metrics compared to both baselines.
['[BOLD] Model', '[BOLD] Dev  [BOLD] MRR', '[BOLD] Dev  [BOLD] P', '[BOLD] Dev  [BOLD] R', '[BOLD] Dev  [BOLD] F1', '[BOLD] Test  [BOLD] MRR', '[BOLD] Test  [BOLD] P', '[BOLD] Test  [BOLD] R', '[BOLD] Test  [BOLD] F1']
It is worth mentioning that our model, denoted as LabelGCN, is trained using the unlicensed training set which is smaller than the one used by compared baselines. Even though our model significantly outperforms the baselines, for a fair comparison, we first test our model using the same decision threshold (0.5) used by previous models. In terms of F1, our best model (LabelGCN) outperforms existing methods by a large margin. Compared to Choi et al. , our model improves on both precision and recall significantly. Compared to the AttentiveNER trained with standard BCE loss, our model achieves much higher recall but performs worse in precision. With higher recalls or more retrieved types, achieving high precision requires being accurate on fine-grained types, which are often harder to predict. As the precision and recall scores both rely on the decision threshold, different models or different metrics can have different optimal thresholds. We also see that recall is usually lagging behind precision on this dataset, indicating that F1 score is mainly affected by the recall and tuning towards recall can usually lead to higher F1 scores. These data points are based on the validation performance given by 50 equal-interval thresholds between 0 and 1. We can see there is a clear margin between our model and the multi-task baseline method (LabelGCN vs Choi et al.) . We can clearly see that the graph convolution layer is the most essential component. The information provided by word embedding is useful and can further improve both precision and recall. In term of this, we recommend future research to use PR curves for more accurate model analysis.