Data-to-text Generation with Entity Modeling
Table 2: Evaluation on RotoWire (RW) and MLB test sets using relation generation (RG) count and precision, content selection (CS) precision and recall, content ordering (CO) in normalized Damerau-Levenshtein distance, and BLEU.
['RW', 'RG', 'RG', 'CS', 'CS', 'CO', 'BLEU']
We report results for our dynamic entity memory model (ENT), the best system of \newciteD17-1239 (WS-2017) which is an encoder-decoder model with conditional copy, and NCP+CC Puduppully et al. We see that ENT achieves scores comparable to NCP+CC, but performs better on the metrics of RG precision, CS precision, and CO. ENT achieves substantially higher scores in CS precision compared to WS-2017 and NCP+CC, without any planning component; CS recall is worse for ENT compared to NCP+CC mainly because the latter model is trained to first create a content plan with good coverage of what to say. Note that ED+CC is a reimplementation of Wiseman et al. ’s (2017) encoder-decoder model (with conditional copy) on MLB. We see that ENT achieves highest BLEU amongst all models and highest CS recall and RG count amongst neural models. The RG precision of ENT is lower than ED+CC. Inspection of model output revealed that on MLB, ED+CC tends to focus on one or two players getting most of the facts about them right, whereas ENT sometimes gets the coreference wrong, and thus lower RG precision. The TEMPL system scores highest on RG precision and count, and CS recall on both datasets. This is because TEMPL can make use of domain knowledge which is not available to the neural models. This suggests that the task of creating MLB game summaries is hard, even for a template system which does not perform any sophisticated generation.