What’s in a Name? Are BERT Named Entity Representations just as Good for any other Name?
Table 4: Adversarial Evaluation of Span-BERT on different tasks
['[EMPTY]', 'NLI (ORG) F1', 'CoRef (PER) F1']
Finally, we explore if BERT’s single token masking model is unfavorable to robust entity representations by comparing with a language model pre-trained by masking spans covering multiple tokens. Specifically, we use Span-BERT Joshi et al. We were surprised that Span-BERT does not provide any better robustness, although it does provide consistent higher accuracy on all tasks. Various metrics such as – the difference between worst and best accuracy, stability are both very similar for BERT and Span-BERT.