Evaluation Benchmarks and Learning Criteriafor Discourse-Aware Sentence Representations
Table 3: Average of the layer number for the best layers in SentEval and DiscoEval.
['[EMPTY]', 'ELMo', 'BERT-Base']
To investigate the performance of individual hidden layers, we evaluate ELMo and BERT on both SentEval and DiscoEval using each hidden layer. For ELMo, we use the averaged vector from the targeted layer. For BERT-Base, we use the vector from the position of the “[CLS]” token. We note that for better visualization, colors in each column are standardized. On SentEval, BERT-Base performs better with shallow layers on USS, SSS, and Probing (though not on SC), but on DiscoEval, the results using BERT-Base gradually increase with deeper layers. From the table, we can see that DiscoEval requires deeper layers to achieve better performance. We assume this is because deeper layers can capture higher-level structure, which aligns with the information needed to solve the discourse tasks.