Deep Learning for Answer Sentence Selection
Table 3: Overview of results on the QA answer selection task. Baseline models are taken from Table 2 in Yih et al. [22]. We also include the results of our best models (bigram + count), which match the current state of the art.
['System  [BOLD] Baselines', 'System  [BOLD] Baselines', 'MAP', 'MRR']
The first model randomly assigns scores to each answer. The second model counts the number of words co-occurring in each QA pair, with another version of that baseline weighting these word counts by IDF values. on MRR. Considering the lack of complexity of our models compared to those of Yih et al., these results are very promising and indicate the soundness of our approach to the QA answer selection task. However, it is also important to then establish the opposite fact, namely that distributional semantics improve over purely word counting model. In fact, this addition boosts both MAP and MRR scores by approximately 10%. We analysed this effect by considering sentences where our combined model (bigram + count) performs better than the counting baseline. Here are two examples, where the model of co-occurrence word count failed to identify the correct answer, but the combined model prevailed: