Learning Better Universal Representations from Pre-trained Contextualized Language Models
Table 2: Performance of our model and baseline model on word, phrase and sentence similarity tasks.”SemEval” stands for the SemEval Task 5(a). The subscript indicates the pooling strategy used to obtain fixed length representations. The best results are in bold. Underlined cells shows tasks where our model outperforms all sentence representation models.
['Method', 'Setences STS12', 'Setences STS13', 'Setences STS14', 'Setences STS15', 'Setences STS16', 'Setences STS B', 'Setences SICK-R', 'Setences Avg.', 'Phrases PPDB', 'Phrases SemEval', 'Words SimLex', 'Words WS-sim', 'Words WS-rel', 'Words MEN', 'Words SCWS', 'Words Avg.']
the STS tasks 2012-2016 For our model and baseline models, we encode each sentence into a fixed length vector with their corresponding encoders and pooling layers, and then compute cosine similarity between each sentence pairs. For the STS benchmark and the SICK-Relatedness dataset, the model is first trained as a regression task and then evaluated on the test set. Note the inconsistent behavior among models on different datasets since the two datasets are distributed differently. Sequences in SemEval2013 Task 5(a) are either single words or two-word phrases. Therefore, Glove and FastText perform better against SkipThought and GenSen on that dataset. Fine-tuning the pre-trained BERTBASE results in higher accuracy than unsupervised models like GloVe, FastText and SkipThought, and is comparable to InferSent and the general-purpose sentence encoder. By first training BERTBASE using a large amount of supervised data, then fine-tuning on SemEval training data, our model yields the highest accuracy. On PPDB where most phrases consist more than two words, RNNs and Transformers are more preferable than simply averaging word embeddings. Especially, our model obtains the highest correlation of 70.64. As mentioned in Faruqui et al. pairs of words that are similar, and pairs of words that are related. For example, ”food” and ”fruit” are similar while ”computer” and ”keyboard” are related. The newly constructed dataset SimLex aim to explicitly quantifies similarity rather than relatedness, where similar words are annotated with higher scores and related words are considered dissimilar with lower scores. The numbers of word pairs in SimLex, WS-353, MEN and SCWS are 999, 353, 3000 and 2003, respectively. Cosine similarity between word vectors are computed and Spearman’s correlation is used for evaluation.