Evaluation Benchmarks and Learning Criteriafor Discourse-Aware Sentence Representations
Table 6: Accuracies (%) for baseline encoder on Sentence Position task when using downstream classifier with or without context.
['Random', '20']
Since our baseline model is already trained with NSP, it is expected to see improvements over a random baseline. The further improvement from using surrounding sentences demonstrates that the context information is helpful in determining the sentence position.