ToTTo: A Controlled Table-To-Text Generation Dataset
Table 7: Performance compared to multiple references on the test set for the subtable input format with metadata.
['Model', 'Overall  [BOLD] BLEU', 'Overall  [BOLD] PARENT', 'Overlap Subset  [BOLD] BLEU', 'Overlap Subset  [BOLD] PARENT', 'Nonoverlap Subset  [BOLD] BLEU', 'Nonoverlap Subset  [BOLD] PARENT']
We present baseline results on ToTTo by examining three existing state-of-the-art approaches: BERT-to-BERT (rothe2019leveraging): A Transformer encoder-decoder model (vaswani2017attention) where the encoder and decoder are both initialized with BERT (devlin2018bert). The original BERT model is pre-trained with both Wikipedia and the Books corpus (zhu2015aligning), the former of which contains our (unrevised) test targets. Thus, we also pre-train a version of BERT on the Books corpus only, which we consider a more correct baseline. Pointer-Generator (see2017get): A Seq2Seq model with attention and copy mechanism (our implementation). puduppully2019data: A Seq2Seq model with an explicit content selection and planning mechanism designed for data-to-text. Both the BERT-to-BERT models perform the best, followed by the pointer generator model. We see that for all models the performance on the non-overlap set is significantly lower than that of the overlap set, indicating that slice of our data poses significant challenges for machine learning models. We also observe that the baseline that separates content selection and planning performs quite poorly. We attest this to the fact that it is engineered to the Rotowire data format with fixed size tables and predefined column names. Fluency - A candidate sentence is fluent if it is grammatical and natural. The three choices are Fluent, Mostly Fluent, Not Fluent. Faithfulness (Precision) - A candidate sentence is considered faithful if all pieces of information are supported by either the table or one of the references. Any piece of unsupported information makes the candidate unfaithful. Covered Cells (Recall) - The percentage of highlighted cells the candidate sentence covers. Coverage with Respect to Reference (Recall) - We ask whether the candidate is strictly more or less informative than each reference (or neither, which is referred to as neutral).