Improving Conditioning in Context-Aware Sequence to Sequence Models
Table 11: Backward PPL on Source
['Model', 'bw PPL', '[ITALIC] Uctx']
In this Section we analyze the different effects of the two types of data augmentations at targeting the challenges facing the query-context conditioned generation. We remove either of the S-T or the S-C data augmentations, and check how that affects the model’s ability at conditioning on the query and utilizing the document. We check the quantitative measures of bw PPL and Uctx on the ELI5 QA task after removing either of the data augmentations. Removing the S-T data augmentation hurts the bw PPL the most, which indicates that sampling S-T data helps the model to generate targets that better adress the query S. On the other hand, removing the S-C data augmentation significantly reduces the context utilization Uctx, which indicates that sampling S-C is critical to utilizing the context effectively.