Can Embeddings Adequately Represent Medical Terminology?New Large-Scale Medical Term Similarity Datasets Have the Answer!Please refer to this version for up-to-date experimental results.
Table 3: Datasets evaluation: term pairs without ground truth score (NaN), Krippendorff’s α IAA, acc(uracy), rec(all), and prec(ision) between ground truth and dataset scores for e(asy)/h(ard) datasets with R(andom)/L(evenshtein) negative sampling.
['[EMPTY]', 'FSN-syn.', 'syn.-syn.', 'poss.-equiv.-to', 'replaced-by', 'same-as']
The difficulty (regarding human judgement) of each dataset is measured in terms of the doctors’ inter-annotator agreement (IAA) and the amount of disagreement (NaNs). The doctors’ decisions can thus be considered reliable. The mostly higher IAA for easy datasets compared to hard ones confirms the intended difficulty difference. Importantly, there is no notable difficulty difference between the two strategies for creating negative instances, so even negative instances with lexically very similar terms can be easily identified as negative by the doctors due to their semantic dissimilarity. Replaced-by datasets are the most difficult as doctors frequently disagree on the similarity of two terms. The quality of datasets is given by the accuracy of the automatically created dataset scores with regards to the ground truth scores. The accuracy of Replaced-by datasets is lower than for the other datasets, but even the lowest accuracy of 0.86 indicates that they are good-quality datasets. We observe that all datasets using the random strategy exhibit a recall of 1, meaning that negative instances in these datasets are indeed dissimilar terms. In contrast, negative instances created using the Levenshtein strategy are sometimes so close that doctors indicate them as in fact being similar terms. The precision furthermore shows that some positive term pairs are in fact dissimilar according to the doctors, which occurs more frequently for the hard datasets. The lower precision in the Possibly-equivalent-to and replaced-by datasets indicates that, as is to be expected, positive instances in these datasets do not always denote exactly the same. An example is the pair of terms ‘Abortion in first trimester’ and ‘Induced termination of pregnancy’, which are related but according to the doctors not the same.