Empirical Evaluation of Pretraining Strategies for Supervised Entity Linking
Table 4: Impact of the candidate selection method on development performance on CoNLL and TAC-KBP 2010. Unsurprisingly, pretraining methods that are closer to the final task setup perform better: For CoNLL, we emphasize the importance of phrase table candidates in pre-training to emulate the use of the alias table , whereas for TAC-KBP, setups that use random candidates are more successful as they are closer to the full classification setup used in this task.
['Candidates source', 'CoNLL No fine-tuning', 'CoNLL Fine-tuned', 'TAC-KBP No fine-tuning', 'TAC-KBP Fine-tuned']
On CoNLL, where we do use an alias table for evaluation, we find that our candidate selection heuristic seems to help the model in pre-training, achieving better performance than any of our ablations. Training with lexically related (phrase table) candidates is particularly important, as this is similar to the disambiguation task the model has to perform. Page candidates are semantically related but generally not lexically related and thus do not bring the same benefits. In fact, they might even distort the distribution of negative candidates as the model performs worse in this setting than with random candidates.