Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models
Table 5: The ablation analysis shows the effects of different k values (ie. window sizes) in our objective, removing the MLM objective during pretraining and training with a small transformer encoder.
['Model', 'SP', 'BSO', 'DC', 'SSP', 'PDTB-E', 'PDTB-I', 'RST-DT', 'avg.']
We observe that using a window size for our objective that is larger than 1 is key to seeing downstream improvements. We believe that this is due to the objective being harder for the model because there is more variation farther from the anchor. At the same time, increasing the window size beyond 2 seems to result in similar performance. This may be because larger distances from the anchor also lead to more ambiguity.