Analysing Data-To-Text Generation Benchmarks
Table 1: Size is the number of instances in the data-set, (Nb. of Attr.) is the number of distinct attributes of the underlying ontology domain, (Entities) is the number of distinct topic entities described in each meaning representation, (Attr.Values) is the number of attribute-value pairs, (MR Len.) is the average length of the input meaning representations computed as the number of attribute value pairs, (MR Ptns) is the number of distinct attribute combinations, namely MR patterns, (MR Inst.) is the number of distinct meaning representations and (PPxMR Inst.) the average (min/max) number of paraphrases per meaning representation. ‡Note that the number of entities is an approximation. We consider entities as distinct entities those given by the name attributes and for the RNNLG data-sets not all dialogue acts give entity descriptions. For instance, inform_count or ?select .
['Dataset', 'Size', 'Nb. of Attr.', 'Entities‡', 'Attr.Values', 'MR Len.', 'MR Ptns', 'MR Inst.', 'PPxMR Inst.']
It shows marked differences in terms of size ( WikiBio16317 being the largest and ImageDesc the smallest), number of distinct relations (from 16 for ImageDesc to 2367 for WikiBio16317 ) and average number of paraphrases (15.11 for ImageDesc against 1 to 3.72 for the other two data-sets). The number of distinct meaning representations (semantic variability) also varies widely (from 77 distinct MRs for the ImageDesc corpus to 12527 for RNNLGLaptop). Overall though, the main observation is that the number of distinct attributes is relatively small and that the high number of distinct meaning representations essentially stems from various combinations of a restricted number of attributes. Overall, the WikiBio data-set, even when restricted to a single type of entity (namely, astronauts) has a higher MSTTR. This higher lexical variation is probably due to the fact that this data-set also has the highest number of attributes (cf. And indeed, there is a positive correlation between the number of attributes in the data-set and MSTTR (Spearman’s rank correlation coefficient rho = +0.385). Third, the choice of domain seems to impact syntactic complexity. Thus the two data-sets in the restaurant domain, although crowd-sourced using different methods, have a similar distribution. Conversely, the various RNNLG data-sets, although developed using the same method, display different distributions. The mean D-level also seems to be impacted by the length of the meaning representation. Thus, in the RNNLG data-sets where the length of the meaning representations (MR) varies (cf. , the Spearman’s rank correlation between MR length and D-level is +0.9 with p=0.037.