Can Embeddings Adequately Represent Medical Terminology?New Large-Scale Medical Term Similarity Datasets Have the Answer!Please refer to this version for up-to-date experimental results.
Table 2: Our new datasets, respective number of (positive & negative) term pairs (Size), average Levenshtein distance of the Pos(itive) and Neg(ative) instances with the R(andom) and L(evenshtein) strategies.
['[BOLD] Dataset', '[BOLD] Size', '[BOLD] Pos', '[BOLD] Neg-R', '[BOLD] Neg-L']
Easy vs. Hard Datasets. The extracted positive instances are expected to all be semantically similar terms. However, lexically the terms can be very similar, e.g. ‘Sacrum sprain’ and ‘Sacral sprain’, or completely different, e.g. ‘Malaria’ and ‘Paludism’. The latter requires a much deeper understanding of medical terminology, whereas the former can be guessed from the surface similarity. To investigate how deep the understanding of term representation models is, we split the positive instances of each dataset into easy and hard ones. The difficulty is measured in terms of Levenshtein distance between the two terms. We experimentally choose a threshold of 5, so that the hard splits mainly contain term pairs with fundamentally different words. SNOMED CT explicitly specifies similar terms (e.g. synonyms), but not dissimilar ones. Naïvely, we can thus consider all term pairs not explicitly specified as similar to be dissimilar. For each dataset, our random negative sampling strategy therefore matches the first term of each positive instance to a randomly selected term from another instance. This may make it easy for models to correctly identify these term pairs as dissimilar. To test if models in fact have a deep understanding of medical terminology, we apply a second negative sampling strategy to create more difficult negative instances: the first term of each positive instance is matched to the term with closest Levenshtein distance that is not (directly or indirectly) specified to be similar. In the hard datasets, the Levenshtein distance of negative instances is even lower than that of positive ones. Thus, for the hard datasets with Levenshtein negative sampling, lexical similarity between terms will not help at all to distinguish similar and dissimilar pairs. Easy vs. Hard Datasets. For the hard datasets, accuracy is much lower than for the easy datasets, sometimes barely over 50% indicating no separation between similar and dissimilar term pairs. In contrast, for the easy datasets, where similar terms have a lower Levenshtein distance than dissimilar terms, the performance of ELMoPubMed is en par with the performance of some of the medical word embeddings. This behaviour can be attributed to the fact that contextual embeddings are based on n-grams/characters, so that lexically similar medical terms are represented by similar vectors.