Evaluating the word-expert approach for Named-Entity Disambiguation
Table 13: Realized performance and oracle skylines — on both news and web portions of development data — as micro-averages, for the LNRM, FUZZ and HEUR dictionaries, as well as for LEX and SENSE supervised classifier variants (both also based on HEUR).
['[EMPTY]', '[ITALIC] news  [ITALIC] realized', '[ITALIC] news  [ITALIC] oracle', '[ITALIC] web  [ITALIC] realized', '[ITALIC] web  [ITALIC] oracle']
An important function of the filtering dictionary is to provide sets of plausible entities, which determine construction of (distantly) supervised training data. Given an ideal classifier, we would prefer to use comprehensive dictionaries that might contain the correct entity for all strings in the evaluation set, even if that meant dragging in many incorrect candidates too. For LNRM and HEUR cascades, gold standard entities are among the dictionaries’ suggestions about 92% of the time (98% for web data). Supervised classification with the LEX strategy lowers these bounds slightly, because in some cases there are no training examples available for the gold entity, which precludes our classifiers from returning the correct result; by expanding the pool of training instances, the SENSE strategy restores this bound to nearly what it was for HEUR. Naturally, higher bounds aren’t necessarily superior: for example, FUZZ yields the overall largest number of possibilities, but has lower realized performance than HEUR.