Incremental Sense Weight Training for theInterpretation of Contextualized Word Embeddings
Table 3: Correlation coefficient test results ρ for original and masked word embeddings with Nmasked (avg. number of dimensions masked out). The BERT embeddings are from the second output layer, and the ELMo and the Flair models from the last output layer.
['[BOLD] Model', '[BOLD] Dim', '[ITALIC] N [BOLD] masked', '[ITALIC] ρ [BOLD] original', '[ITALIC] ρ [BOLD] masked']
The number of dimensions masked is averaged throughout all sense groups. For ELMo and Flair, with the insignificant embedding dimensions masked out, the sense groups show a better correlation score. For the ELMo models, the number of embeddings that can be discarded increases with the distance of the output layer to the input layer. This result corresponds to ELMo’s claim that the embeddings with output layers closer to the input layer are semantically richer Peters et al.