Rationalizing Medical Relation Prediction from Corpus-level Statistics
Table 3: Human evaluation on the quality of rationales.
['[EMPTY]', 'OWA Rationales', 'CWA Rationales']
Expert Evaluation. We first explained to the physician about the recall and recognition process in our framework and how model rationales are developed. They endorsed such reasoning process as one possible way to gain their trust in the model. Next, for each target pair for which our model correctly makes the prediction, they were shown the top-5 rationales produced by our framework and were asked whether each rationale helps them better trust the model prediction. For each rationale, they were asked to score it from 0 to 3 in which 0 is no helpful, 1 is a little helpful, 2 is helpful and 3 is very helpful. In addition to the individual rationale evaluation, we further compare the overall quality of CWA and OWA rationales, by letting experts rank them based the helpfulness of each set of rationales (the rationale set ranked higher gets 1 ranking score and both get 0 if they have the same rank). We refer readers to the appendix for more details of the evaluation protocol. Out of 30, OWA wins in 17 cases and gets higher scores on individual rationales per case on average. To get a better idea of how the OWA model obtains more trust, we calculate the average sum score per case, which shows the OWA model gets a higher overall score per case. Considering in some cases only a few rationales are able to get non-zero scores, we also calculate the average max score per case, which shows that our OWA model generally provides one helpful rationale (score>2) per case. Overall, as we can see, the OWA rationales are more helpful to gain expert trust.