ViGGO: A Video Game Corpus for Data-To-Text Generationin Open-Domain Conversation
Table 4: Baseline system performance on the ViGGO test set. Despite individual models (Bo3 – best of 3 experiments) often having better overall scores, we consider the Ao3 (average of 3) results the most objective.
['[EMPTY]', '[BOLD] BLEU', '[BOLD] METEOR', '[BOLD] ROUGE', '[BOLD] CIDEr', '[BOLD] SER']
Automatic Metrics We evaluate our model’s performance on the ViGGO dataset using the following standard NLG metrics: BLEU Papineni et al. Additionally, with our heuristic slot error rate (SER) metric we approximate the percentage of failed slot realizations (i.e., missed, incorrect, or hallucinated) across the test set. Since our baseline model does not implement any form of a copy mechanism, it fails on instances with out-of-vocabulary terms, such as the values of the specifier slot in the test set. Therefore, more robust models have good potential for improving on our scores.