Improving Conditioning in Context-Aware Sequence to Sequence Models
Table 3: Results on the ELI5 dataset.
['Model', 'R-1', 'R-2', 'R-L', 'PPL']
We report results for the sequential baseline and the author’s multitask system. The latter is trained on 10 different tasks, including predicting using all combinations of S, C, and T as input and output, as well as a BERT-like [bert] masked word prediction task. This leads to a significant improvement in both rouge and perplexity. Finally, since the model is only tested on the setting where the model predicts the target T given the source S and context C, we fine-tune a multitask-trained model using both the standard sequential approach and our interleave setting. First, all of our proposed approaches always out-perform the standard sequential seq2seq training as well as the end2end and doc-NMT models on the Wizards of Wikipedia and document-level MT respectively. These results validate the importance of the data augmentation method, as well as the separate source and context encoding with focused context attention. Between the three proposed architectures, interleave consistently outperforms concatenate and alternate, with a slightly higher perplexity on Wizards of Wikipedia. This is especially notable as it is the architecture with the least number of operations, since each layer only attends to one of the sources. On the ELI5 dataset, interleave still lags a little behind multitask in terms of perplexity but achieves better rouge, which, according to [eli5], correlates better with human judgements. Note also that while our data augmentation method randomly samples a different input and output setting in up to 50% of cases, the multitask setting has to go through 10 different versions of each example, leading to much slower convergence (24 hours for interleave, 5 days for multitask). Further, since the interleave architecture uses the same set of parameters as the standard encoder-decoder transformer, we can easily fine-tune a trained multitask model using our framework. First, we note that the multitask model performs better than sequential on both metrics, which is coherent with the findings from [eli5]’s human evaluation of the model outputs, and validates the use of the proposed metrics. Additionally, the difference in Uctx appears to be the more significant of the two, indicating that our method especially improves over multitask in terms of making good use of the context.