S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension
Table 2: The performance on the MS-MARCO test set. *Using the ensemble result of extraction models as the input of the synthesis model.
['[BOLD] Method', '[BOLD] ROUGE-L', '[BOLD] BLEU-1']
Our extraction model achieves 41.45 and 44.08 in terms of ROUGE-L and BLEU-1, respectively. We sum the probability at each position of each single model to decide the ensemble result. Finally we select 13 models for ensemble, which achieves 42.92 and 44.97 in terms of ROUGE-L and BLEU-1, respectively, which achieves the state-of-the-art results of the extraction model. Then we test our synthesis model based on the extracted evidence. Our synthesis model achieves 3.78% and 3.73% improvement on the single model and ensemble model in terms of ROUGE-L, respectively. Our best result achieves 46.65 in terms of ROUGE-L and 44.78 in terms of BLEU-1, which outperforms all existing methods with a large margin and are very close to human performance. Moreover, we observe that our method only achieves significant improvement in terms of ROUGE-L compared with our baseline. The reason is that our synthesis model works better when the answer is short, which almost has no effect on BLEU as it is normalized with all questions.