Unsupervised Natural Language Generation with Denoising Autoencoders
Table 5: Human evaluation results: We generated 279 output sequences for each of the 3 listed systems. Each sequence has been evaluated by 3 different raters and the score is the average of 837 ratings per system. For each task and sequence, the raters where asked to give a score between 0 and 5. A score of 5 for fluency means that the text is fluent and grammatical correct. A score of 5 for all information means that all information from the structured data is mentioned. A score of 0 for extra/ false information means that no information besides the structured data is mentioned in the sequence. Scores labeled with † are significant better than all other systems (p<0.0001).
['system', 'fluency', 'all', 'extra/ false']
Each task has been given to three different raters. Consequently, each output has a separate score for each question that is the average of 3 different ratings. We included the two supervised baselines and our best unsupervised setup in the human evaluation. The unsupervised setup outperforms the supervised setups in fluency. One explanation is that our unsupervised system includes additional unlabeled data that can not be included in a supervised setup. Due to our unsupervised learning approach that all words in the structured data need to be included in the final output, the unsupervised system did not miss any information. Further, all three outputs included little false or wrong information that was not included in the structured data. All in all the output of the unsupervised system is better than the two supervised systems. showed that our novel unsupervised approach is significantly better than both baselines regarding fluency and mentioning all information with the likelihood of incorrectly rejecting the null hypothesis of p<0.0001.