ColdGANs: Taming Language GANs with Cautious Sampling Strategies
Figure 4: Probability that the generated text is human according to Dϕ on CNN/DM.
['[EMPTY]', 'Fluency', 'Relevance', 'Answerability']
Therefore, we additionally conducted a human evaluation on the QG task. Three professional English speakers were asked to judge, on a 1-to-5 Likert scale, to what extent the generated questions were: well-posed and natural (Fluency), relevant to their context (Relevance), and answerable, by looking at their context and answer (Answerability). A similar result was reported by yoon2020learning (refer to Table 2 in their paper). A plausible explanation is that humans are more inclined to use informal language and make grammar mistakes. For instance the human question ”About how many yellow cabs operate in New York?” sounds slightly less formal than the one, generated by ColdGAN, ”How many yellow taxicabs are in Manhattan ?”. Compared to MLE, ColdGAN enables to significantly improve in term of fluency, while remaining competitive on other metrics, consistently with our experiments on exposure bias.