Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models
Table 1: Conpono improves the previous state-of-the-art on four DiscoEval tasks. The average accuracy across all tasks is also a new state-of-the-art, despite a small drop in accuracy for PDTB-E. BERT-Base and BERT-Large numbers are reported from Chen et al. (2019), while the rest were collected for this paper. We report standard deviations by running the evaluations 10 times with different seeds for the same Conpono model weights.
['Model', 'SP', 'BSO', 'DC', 'SSP', 'PDTB-E', 'PDTB-I', 'RST-DT', 'avg.']
We report results from versions of Conpono using each of these encoding approaches, labeled isolated to represent separate encoding and uni-encoder to represent joint encoding of the anchor and target without a separate anchor encoding. Results: Our model excels in particular on the sentence ordering and coherence tasks (SP, BSO, and DC). Note that our model parameter count is the same as BERT-Base but it outperforms BERT-Large, which has significantly more parameters and has used much more compute for pretraining. BERT-Base BSO scores tend to fall between those of BERT-Base and our model, implying that the sentence ordering objective is improving the models for this benchmark, but that binary sentence ordering is not sufficient to capture the added benefits of including more fine-grained ordering and negative examples.