Can Embeddings Adequately Represent Medical Terminology?New Large-Scale Medical Term Similarity Datasets Have the Answer!Please refer to this version for up-to-date experimental results.
Table C.3: AUC of each embedding (fJ for word embeddings, τ for contextual embeddings) for datasets created with random negative sampling.
['Dataset', '[BOLD] FSN-syn.  [BOLD] easy', '[BOLD] FSN-syn.  [BOLD] hard', '[BOLD] syn-syn.  [BOLD] easy', '[BOLD] syn-syn.  [BOLD] hard', '[BOLD] poss.-equ.  [BOLD] easy', '[BOLD] poss.-equ.  [BOLD] hard', '[BOLD] repl.-by  [BOLD] easy', '[BOLD] repl.-by  [BOLD] hard', '[BOLD] same-as  [BOLD] easy', '[BOLD] same-as  [BOLD] hard']
Since our new datasets frame a binary classification task, we evaluate the embeddings’ separability of similar versus dissimilar term pairs using the area under the ROC curve (AUC) and accuracy based on a classification threshold optimising the accuracy (different threshold for each embedding and similarity metric). Significance between classifications of the different embeddings using the optimised thresholds is measured by McNemar’s test. Since the accuracy scores follow the AUC trends (see Embedding Comparison. MeSH intr yields the best overall separation of similar and dissimilar term pairs, significantly outperforming the non-medical word embeddings and the contextual embeddings as well as most of the medical word embeddings – especially on the hard datasets. In contrast, the performances of all medical word embeddings on the datasets with random negative sampling are very similar and high This shows that, as is to be expected, random negative sampling creates term pairs that are easily identifiable as dissimilar. In the following, we thus focus on the datasets with Levenshtein negative sampling. It can be observed that all accuracy scores are very high, illustrating that similar concept pairs in these datasets are easy to sepearte from dissimilar ones. The low AUC for the hard datasets in the first table indicate the high difficulty of these datasets, for which none of the embeddings is able to distinguish similar and dissimilar concept pairs.