Zero-Shot Entity Linking by Reading Entity Descriptions
(a)
['Pretraining', 'W1tgt', 'W2tgt', 'W3tgt', 'W4tgt', 'Avg']
DAP improves all pre-training strategies with an additional pre-training stage on only target-domain data. The best setting, UWB→Usrc+tgt→Utgt, chains together all existing strategies. DAP improves the performance over a strong pre-trained model Devlin et al. To further analyze the results of DAP, we plot the relationships between the accuracy of Masked LM (MLM accuracy) on target unlabeled data and the final target Adding an additional pre-training stage on the target unlabeled data unsurprisingly improves the MLM accuracy. More interestingly, we find that improvements in MLM accuracy are consistently followed by improvements in entity linking accuracy. It is intuitive that performance on unsupervised objectives reflect the quality of learned representations and correlate well with downstream performance. We show empirically that this trend holds for a variety of pre-training strategies.