Inferring Which Medical Treatments Work fromReports of Clinical Trials
Table 6: Average results achieved when models are provided directly with the reference evidence spans.
['Model', 'Precision', 'Recall', 'F1']
Access to this oracle yields a 20+ point jump in F1, indicating that accurately extracting the relevant evidence is critical. we attempt to elucidate how well (or poorly) attention mechanisms fail to find supporting evidence. This suggests that the model is failing to sufficiently attend to the relevant evidence. This shows the total evidence mass realized by models on the validation set (after training for the final task of evidence inference). Even pretrained models assign <14% of total attention mass to actual evidence tokens. Further, ignoring the article entirely (relying only on the prompt) degrades performance by only ∼5 points in F1, again suggesting that even the pretrained, conditioned attention variant is not making good use of the relevant evidence contained in articles. The evidence token mass metrics also support this: The best models we have proposed consistently place only ∼10-15% of the attention mass on tokens actually marked as containing relevant evidence.