Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking
Table 2: Investigating the types of strong precision errors of BERT+Entity trained in Setting I on CoNLL’03/AIDA (testa) on 100 randomly sampled strong precision errors from the validation dataset.
['Reason for error', '#']
The largest source of errors was that BERT+Entity did predict Nil instead of an entity. We hypothesized that most of the no prediction errors are because those entities have only a low frequency in the training data, i.e. this could be solved by increasing the model size and improving the training time. Another source of error we observed was that the context size was too small due to the fragment size. A surprisingly positive result from the error analysis was that in only 3% a wrong span caused the error. Motivated by the observations we devised the follow-up experiment Setting II (see