Table-to-Text Generation with Effective Hierarchical Encoder on Three Dimensions (Row, Column and Time)
Table 3: Human evaluation results.
['[BOLD] Model', '[BOLD] #Sup', '[BOLD] #Cont', '[BOLD] #Gram', '[BOLD] #Coher', '[BOLD] #Conc']
Following human evaluation settings in \newcitePuduppully, we conducted the following human evaluation experiments at the same scale. The second experiment is to assess whether the improvement on relation generation metric reported in automatic evaluation is supported by human evaluation. We compared our full model with gold texts, template-based system, CC Wiseman et al. Puduppully et al. We randomly sampled 30 examples from test set. Then, we randomly sampled 4 sentences from each model’s output for each example. We provided the raters of those sampled sentences with the corresponding NBA game statistics. They were asked to count the number of supporting and contradicting facts in each sentence. Each sentence is rated independently. Unsurprisingly, template-based system includes most supporting facts and least contradicting facts in its texts because the template consists of a large number of facts and all of those facts are extracted from the table. Also, our model produces less contradicting facts than other two neural models. Although our model produces less supporting facts than NCP and CC, it still includes enough supporting facts (slightly more than gold texts). Also, comparing to NCP+CC (NCP) ’s tendency to include vast information that contain redundant information, our model’s ability to select and accurately convey information is better. All other results (Gold, CC, NCP and ours) are significantly different from template-based system’s results in terms of number of supporting facts according to one-way ANOVA with posthoc Tukey HSD tests. All significance difference reported in this paper are less than 0.05. Our model is also significantly different from the NCP model. As for average number of contradicting facts, our model is significantly different from other two neural models. Surprisingly, gold texts were found containing contradicting facts. We checked the raters’s result and found that gold texts occasionally include wrong field-goal or three-point percent or wrong points difference between the winner and the defeated team. We can treat the average contradicting facts number of gold texts as a lower bound.