Machine Translation Pre-training for Data-to-Text Generation - A Case Study in Czech
Table 8: Experiments with low-resource NLG
['Training Size', 'Model', 'BLEU ↑', 'SER ↓']
Next, we fine-tune each of these models on the data-to-text task. Both the high and medium resource models have a comparable SER. Even the low resource model, pre-trained on just 1% of the translation corpora is significantly better than mass, which has been pre-trained on almost 1.6 billion tokens. The results indicate that machine translation based transfer learning can be successfully applied even when the size of parallel corpus is small, and thus holds promise for low-resource languages. In this section we study the effects of transfer learning when the size of the fine-tuning corpus is small. We create two random subsets from the NLG training data of size 100 and 1000. We find that once again, NMT offers substantial gains over MASS. When fine-tuning on 1000 examples, pre-training with NMT is substantially better (20% improvement on SER, +3 on BLEU) than fine-tuning MASS with the full dataset. Remarkably, with just 100 examples, our model outperforms training from scratch on the entire training set by over 3 BLEU, while reducing SER by over 30 points. These results lead us to believe that NMT pre-training can lead to substantial cost savings with respect to training data annotation.