Contextual Lensing of Universal Sentence Representations
Table 2: Comparison of USVs on 9 downstream tasks. The first seven tasks are evaluated by training a logistic regression classifier directly on top of the sentence vectors. Performance is accuracy. The last two tasks report Spearman correlation of unsupervised textual similarity. Δ indicates the mean improvement over all tasks with respect to BERT BOW. Best results per column are bolded. Results that are not ours are obtained from Reimers and Gurevych (2019).
['Model', 'MR', 'CR', 'SUBJ', 'MPQA', 'SST', 'TREC', 'MRPC', 'STSb', 'SICK', 'Δ']
Here we make two observations. First, our results outperform on aggregate against existing non-BERT universal sentence encoders while only learning a single weight matrix on top of the contextualized embeddings. Second, while SBERT outperforms on aggregate, the Δ is small relative to a basic BERT BOW baseline. This demonstrates that the gain from fine-tuning all of BERT for these tasks are minimal and much of the performance improvements over a naive BERT BOW baseline can be achieved by adapting fixed contextualized embeddings. (Liu et al., While fine-tuning RoBERTa on static tasks substantially improves over BERT, the gains are minimal for downstream sentence evaluations. Taken together, these results suggest that the fine-tune paradigm that is so crucial for static evaluations may not directly carry over to learning USVs.