Can Embeddings Adequately Represent Medical Terminology?New Large-Scale Medical Term Similarity Datasets Have the Answer!Please refer to this version for up-to-date experimental results.
Table 4: Spearman’s correlation of each embedding (fJ for word embeddings, avg_cos for GloVe, τ for contextual embeddings). An embedding has significantly better/worse correlation than the number of embeddings given by the positive/negative superscripts (α=0.0002, i.e. α=0.05 with Bonferroni correction). MM-av: average scores of coders and physicians.
['Dataset', '[BOLD] Hlia.', '[BOLD] MM-av', '[BOLD] Mayo', '[BOLD] Sim', '[BOLD] Sim-m', '[BOLD] Rel', '[BOLD] Rel-m', '[BOLD] SimLex', '[BOLD] SimVerb']
Embedding Comparison. Overall, the correlations of word embeddings are moderate to strong, suggesting that embeddings are able to decently encode medical terms and their similarity. For the Hliaoutakis and MiniMayo datasets, no significant differences between biomedical and, in most cases, even the non-medical word embeddings are observed, despite correlation differences as large as 0.2. This is due to the small size of these datasets and highlights the need for larger datasets to obtain more meaningful embedding comparisons. On the UMNSRS-Sim/Rel datasets, the BioNLP embeddings perform significantly worse than most other biomedical embeddings, even though they achieve the highest correlations on the very small datasets. This demonstrates that existing datasets do not allow any judgements about the generalisability of embeddings to unseen similarity instances. The other biomedical word embeddings do not exhibit significant differences and even the non-medical word embeddings do not perform significantly worse. This raises the question if existing datasets are representative of the highly difficult medical terminology. All word embeddings significantly outperform all contextual embeddings except ELMoPubMed, which performs significantly better than the other contextual embeddings. BERT models usually require fine tuning, so their lower performance is expected. Flair’s lower performance likely stems from it not having an explicit notion of words, whereas the remaining contextual embeddings lack medical knowledge.