Machine Translation Pre-training for Data-to-Text Generation - A Case Study in Czech
Table 9: Out-of-Vocabulary test set. unique refers to the number of unique values the slot takes in the test set. total is the number of times the slot appears.
['slot', 'unique', 'total', 'errors', 'accuracy']
Unsupervised pre-training through MASS completely fails to generalize to OOV values - none of the slots are realized in the predictions. Looking at the output, we noticed that mass has a strong tendency to hallucinate or simply output slot values seen during training. The poor performance further reinforces the practical popularity of delexicalized models and highlights the need for challenging OOV test sets. binmt on the other hand, remains robust - 91.5% of the slots from the MR are realized in the predictions. The results confirm that NMT can greatly improve the robustness to unseen values. Further reducing the performance gap between seen and unseen values is an important area of future work.