S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension
Table 5: The performance of questions in different levels of necessary of synthesis in terms of ROUGE-L on MS-MARCO development set.
['[BOLD] Category', '[BOLD] Extraction', '[BOLD] Extraction  [BOLD] +Synthesis']
For the question whose answer can be exactly matched in the passage, our answer synthesis model performs slightly worse because the sequence-to-sequence model makes some deviation when copying extracted evidences. In other categories, our synthesis model achieves more or less improvement. For the question whose answer can be almost found in the passage (ROUGE-L≥0.8), our model achieves 0.2 improvement even though the space that can be raised is limited. For the question whose upper performance via answer extraction is between 0.6 and 0.8, our model achieves a large improvement of 2.0. Part of questions in the last category (ROUGE-L<0.2) are the polar questions whose answers are “yes” or “no”. Although the answer is not in the passage or question, our synthesis model can easily solve this problem and determine the correct answer through the extracted evidences, which leads to such improvement in this category. However, in these questions, answers are too short to influence the final score in terms of BLEU because it is normalized in all questions. Moreover, the score decreases due to the penalty of length. Due to the limitation of BLEU, we only report the result in terms of ROUGE-L in our analysis.