Answering Science Exam Questions UsingQuery Reformulation with Background Knowledge
Table 1: Essential terms classification performance as measured by token-level (Acc)uracy, (Pr)ecision, (Re)call, and F1 score. The results for ET Classifier reflect the 70/9/21 train/dev/test split reported in Khashabi et al. (2017). We follow ET Net in using a random 80/10/10 train/dev/test split performed after filtering out questions that appear in the ARC dev/test sets.
['Method', 'Acc', 'Pr', 'Re', 'F1']
ET Classifier Khashabi et al. ET Net Ni et al. ET Classifier trains an SVM using over 120 features based on the dependency parse, semantic features of the sentences, cluster representations of the words, and properties of question words. While the ET Classifier was evaluated using a 79/9/21 train/dev/test split, we follow Ni et al. in using an 80/10/10 split and remove questions from the Essential Terms dataset that appear in the ARC dev/test partitions. The key insights from this experimental evaluation are as follows: NCRF++ significantly outperforms the seq2seq model with respect to all evaluation metrics (see results with GloVe 840B.300d). NCRF++ is competitive with respect to ET Net and ET Classifier (without the heavy feature engineering of the latter system). It has significantly better accuracy and recall than ET Classifier although its F1-score is 3% inferior. When used with CompleX graph embeddings Trouillon et al.