How would you say that? Pictures elicit better NLG data from the crowd.
Table 3: Human evaluation of the data collected with each MR (** = p<0.01 and *** = p<0.001 for Pictorial versus Textual conditions). Italics denote averages across all numbers of attributes.
['[EMPTY]', '[BOLD] Textual MR Mean', '[BOLD] Textual MR StDev', '[BOLD] Pictorial MR Mean', '[BOLD] Pictorial MR StDev']
Informativeness was defined (on the questionnaires) as whether the utterance “provides enough useful information about the venue”. A two-way ANOVA was conducted to examine the effect of MR modality and the number of attributes on the perceived Informativeness. There was no statistically significant interaction between the effects of modality and the number of attributes in the MR, F(2,1236) = 1.79, p=0.17. A main effects analysis showed that the average Informativeness of utterances elicited through the pictorial method (4.51) was significantly higher than that of utterances elicited using the textual/logical modality (4.28), with p<0.01. This is an increase of 0.23 points on the 6-point scale (=4.6%) in average Informativeness rating for the pictorial condition. Naturalness was defined (on the questionnaires) as whether the utterance “could have been produced by a native speaker”. A two-way ANOVA was conducted to examine the effects of MR modality and the number of attributes on the perceived Naturalness. There was no statistically significant interaction between the effects of modality and the number of attributes in the MR, F(2,1236) = 0.73, p=0.48. A main effects analysis showed that the average Naturalness of utterances elicited using the pictorial modality (4.43) was significantly higher than that of utterances elicited using the textual/logical modality (4.09), with p<0.001. This is an increase of about 0.34 points on the scale (=6.8%) for average Naturalness rating for the pictorial condition. A two-way ANOVA was conducted to examine the effect of MR modality and the number of attributes on the perceived Phrasing. There was no statistically significant interaction between the effects of modality and the number of attributes in MR, F(2,1236) = 0.85 , p=0.43. A main effects analysis showed that the average Phrasing score for the utterances elicited using the pictorial modality was significantly higher than that of the utterances elicited using the textual/logical modality, with p<0.001. This is an increase of +0.39 points (about 7.8%) in average Phrasing rating for the pictorial condition.