aNMM: Ranking Short Answer Texts with Attention-Based Neural Matching Model
Table 4: Results of TREC QA on TRAIN and TRAIN-ALL without combining additional features (Compare with deep learning methods).
['Training Data Method', 'TRAIN MAP', 'TRAIN MRR', 'TRAIN-ALL MAP', 'TRAIN-ALL MRR']
Our first experimental setting is ranking answer sentences directly by the predicted score from aNMM without combining any additional features. In this table, we compare the results of aNMM with other previous deep learning methods including CNN [Yu:2014, Severyn:2015: LRS:2766462.2767738] and LSTM [DBLP:conf/acl/WangN15]. We summarize our observations as follows: (1) Both aNMM-1 and aNMM-2 show significant improvements for MAP and MRR on TRAIN and TRAIN-ALL data sets comparing with previous deep learning methods. Specifically, if we compare the results of aNMM-1 with the strongest deep learning baseline method by Severyn et al. [Severyn:2015:LRS:2766462.2767738] based on CNN , we can see aNMM-1 outperform CNN for 14.67% in MAP on TRAIN, 9.15% in MAP on TRAIN-ALL. For MRR, we can also observe similar significant improvements of aNMM-1. These results show that with the value-shared weight scheme instead of the position-shared weight scheme in CNN and term importance learning with question attention network, aNMM can predict ranking scores with much higher accuracy comparing with previous deep learning models for ranking answers. (2) If we compare the results of aNMM-1 and aNMM-2, we can see their results are very close. aNMM-1 has slightly better performance than aNMM-2. This result indicates that adding one more hidden layer to incorporate multiple bin nodes does not necessarily increase the performance for answer ranking in TREC QA data. From the perspective of model efficiency, aNMM-1 could be a better choice since it can be trained much faster with good prediction accuracy. However, for larger training data sets than TREC QA data, aNMM-2 could have better performance since it has more model parameters and is suitable for fitting larger training data set. We leave the study of impact of the number of hidden layers in aNMM to future work.