Natural Language Generation at Scale:A Case Study for Open Domain Question Answering
Table 4: Human evaluation results on three QA NLG datasets with increasingly larger ontologies. The models reported are a baseline with two encoders, for slot types and slot values, and its extensions with a delexicalised or lexicalised previous utterance. We report averages of Naturalness (Nat.), Informativeness (Inf.), and how conversational the response was judged (Conv.) on a scale of 1 to 6. Additionally, we report the average of whether responses could be considered an answer to the given question (Ans.), given to annotators as a binary choice.
['[EMPTY]', '[BOLD] QA.1  [ITALIC]  [BOLD] Nat.', '[BOLD] QA.1  [ITALIC]  [BOLD] Inf.', '[BOLD] QA.1  [ITALIC]  [BOLD] Conv.', '[BOLD] QA.1  [ITALIC]  [BOLD] Ans.', '[BOLD] QA.2  [ITALIC]  [BOLD] Nat.', '[BOLD] QA.2  [ITALIC]  [BOLD] Inf.', '[BOLD] QA.2  [ITALIC]  [BOLD] Conv.', '[BOLD] QA.2  [ITALIC]  [BOLD] Ans.', '[BOLD] QA.3  [ITALIC]  [BOLD] Nat.', '[BOLD] QA.3  [ITALIC]  [BOLD] Inf.', '[BOLD] QA.3  [ITALIC]  [BOLD] Conv.', '[BOLD] QA.3  [ITALIC]  [BOLD] Ans.']
Human evaluation, on the other hand, seems in line with the picture depicted by BLEU and SERtrg. This confirms our hypothesis that SERmr might be a less reliable metric to evaluate NLG QA output. Moreover, although we notice a consistent but not drastic degradation in terms of BLEU and SERtrg in correlation with bigger ontologies, human evaluation shows an even more gentle degradation between QA.1 and 3 for many metrics. Interestingly, it seems the ability of all models to give a proper answer to the question (answer) increases from QA.1 to 3.