Multi-Perspective Context Matching for Machine Comprehension
Table 2: Results on the SQuAD test set. All the results here reflect the SQuAD leaderboard as of Dec. 9, 2016.
['Models Single', 'Models Logistic Regression', 'EM 40.4', 'F1 51.0']
Our single MPCM model achieves the EM of 65.5, and the F1 score of 75.1. We also build an ensemble MPCM model by simply averaging the probability distributions of 5 models, where all the models have the same architecture but initialized with different seeds. With the help of the simple ensemble strategy, our MPCM model improves about 3% in term of EM, and 2% in term of F1 score. Comparing the performance of other models, our MPCM models achieve competitive results in both single and ensemble scenarios. Chunking and Ranking. In this kind of methods, a list of candidate chunks (answers) are extracted firstly. Then, models are trained to rank the correct chunk to the top of the list. However, over 20% of the questions do not have any correct answers within the candidate list. To increase the recall, \newciteyu2016end extracted candidate chunks based on some part-of-speech patterns, which made over 90% of the questions answerable. \newcitelee2016learning enumerated all possible chunks (up to 30-grams) within the passage, learned a fixed length representations for each chunk with a multi-layer BiLSTM model, and scored each chunk based on the fixed length representations. Boundary Identification. Instead of extracting a list of candidate answers, this kind of methods learns to identify the answer span directly. Generally, some kinds of question-aware representations are learnt for each time step of the passage, then the beginning and ending points are predict based on the representations. However, the returned positions are not guaranteed to be consecutive. Their model first captured the interactions between the question and the passage with a co-attentive encoder, then a dynamic pointing decoder was used for predicting the beginning and ending points. \newciteseo2016bidirectional proposed a similar model with \newcitexiong2016dynamic. This model employed a bi-directional attention flow mechanism to achieve a question-aware context representations for the passage, then the beginning and ending points were predict based on the representations. Our model also belongs to this category. However, different from all the previous models, our model generates the question-aware representations by explicitly matching contextual embeddings of the passage with the question from multiple perspectives, and no lexical or word vector information is passed to the boundary identification layer.