Incremental Sense Weight Training for theInterpretation of Contextualized Word Embeddings
Table 2: Results for the original and embeddings with 5% dimensions masked.
['[BOLD] Model', '[BOLD] Original', '[BOLD] Masked']
The embeddings from all output layers of ELMo, BERT and Flair are evaluated. Half the embeddings are improved if being masked using the 5% threshold, especially the last 10 layer outputs. Surprisingly, the last layer output score is boosted by 3%.