Unification-based Reconstruction of Explanations for Science Questions
Table 4: Performance of BERT on answer prediction (test-set) with and without explanations.
['[BOLD] Answer Prediction Model', '[BOLD] Accuracy Easy', '[BOLD] Accuracy Challenge', '[BOLD] Accuracy Overall']
BERT without explanations is provided with question and candidate answer only. On the other hand, BERT with explanation receives the following inputs: the question (q), a candidate answer (ci) and the explanation computed for ci (Ei). In this setting, the model is fine-tuned for binary classification (bertb) to predict a set of probability scores P={p1, p2, ..., pn} for each candidate answer in C={c1, c2, ..., cn}: bertb({[CLS]} || q||ci || {[SEP]} || Ei)=pi (3) The binary classifier operates on the final hidden state corresponding to the [CLS] token. To answer the question q, the module selects the candidate answer ca such that a=argmaxipi. Notably, a significant improvement in accuracy can be observed when BERT is provided with explanations retrieved by the reconstruction modules (+9.78% accuracy with RS BM25 + US BM25 model). The improvement is consistent on the easy split (+6.92%) and particularly significant for challenge questions (+15.69%). Overall, we observe a correlation between more precise explanations and accuracy in answer prediction, with BERT + RS BM25 being outperformed by BERT + RS BM25 + US BM25 for each value of K. Moreover, steadier results in answer prediction adopting the RS + US model suggest a positive contribution from abstract explanatory facts. Additional investigation of this aspect will be a focus for future work.