Fluent Response Generation for Conversational Question Answering
Table 3: Human evaluation results of all the models and baselines on sample of SQuAD-dev-test. In the first three rows B. stands for baseline. In the last row ”(o)” stands for oracle. In Column 3 PPL stands for validation perplexity. All the values are percentage (out of 100) of responses from each model that belong to specific option(a to e) selected by annotators.
['Model', 'Data', 'PPL', 'a', 'b', 'c', 'd', 'e', '[EMPTY]']
However, validation perplexity is a weak evaluator of generation models. Also, due to the lack of human-generated references in SQuAD-dev-test, we cannot use other typical generation based automatic metrics. Therefore, we use Amazon Mechanical Turk to do human evaluation. Each response is judged by 5 annotators. We ask the annotators to identify if the response is conversational and answers the question correctly. While outputting answer-phrase to all questions is trivially correct, this style of response generation seems robotic and unnatural in a prolonged conversation. Therefore, we also ask the annotators to judge if the response is a complete-sentence (e.g. “it is in Indiana”) and not a sentence-fragment (e.g. “Indiana”). For each question and response pair, we show the annotators five options based on the three properties (correctness, grammaticality, and complete-sentence). The best response is a complete-sentence which is grammatical and answers the question correctly (i.e. option e). Other options give us more insights into different models’ behavior. For each response, we assign the majority option selected by the annotators and aggregate their judgments into buckets. Almost all models perform better when trained with SS+ data showing that the additional data from Natural Questions and HarvestingQA is helping. Except for the PGN model trained on SS data, all other variants perform better than STs+BERT baseline in terms of option e. The GPT-2 model trained on SS data from scratch does not perform very well because of the small size of training data. The pretraining with OpenSubtitiles questions boosts its performance (option e % for GPT-2-Pre model variants > option e % for GPT-2 model variants). The best model however is D-GPT when finetuned with SS+ dataset. While retaining the correct answer, it makes less grammatical errors (lower % in option c and d compared to other models). This shows that D-GPT can generate better quality responses with accurate answers.