Inferring Which Medical Treatments Work fromReports of Clinical Trials
Table 5: Average results achieved (macro-averages over five runs) by the neural model when it is provided only the article or only the prompt. We reproduce results for the best model from Table 2 and the vanilla (no attention) end-to-end neural network for context.
['Model', 'Precision', 'Recall', 'F1']
A natural question that arises in NLP tasks in which the output depends on both a document and a question (here, a prompt) is: how much does the latter in fact influence model predictions Kaushik and Lipton Relying only on the prompt (ignoring the article completely) achieves surprisingly strong performance, outperforming a vanilla neural model (sans attention). This is not entirely unreasonable, as certain intervention types will tend to correlate with significant vs insignificant findings, i.e., the prompt itself contains signal. The neural model without attention is likely simply unable to extract meaningful signal from lengthy articles, and so induced representations merely add noise. By contrast, ignoring the prompt severely degrades performance.