Learning Better Universal Representations from Pre-trained Contextualized Language Models
Table 3: Ablation study on training objectives, pooling strategies, concatenation methods and the value of k in negative sampling. Lower layers are initialized with BERTbase. ”NLI”, ”PI” and ”PTC” stands for sentence-level natural language inference, phrase/word-level paraphrase identification and phrase/word-level pairwise text classification tasks, respectively. ”PPDB” represents the phrase-level similarity test set extracted from the PPDB database.
['[EMPTY]', 'Setences STS12', 'Setences STS13', 'Setences STS14', 'Setences STS15', 'Setences STS16', 'Setences STS B', 'Setences SICK-R', 'Setences Avg.', 'Phrases PPDB', 'Phrases SemEval', 'Words SimLex', 'Words WS-sim', 'Words WS-rel', 'Words MEN', 'Words SCWS', 'Words Avg.']
We train our model on different combinations of training objectives to investigate in what aspect does the model benefit from each of them. The phrase and word level tasks using the PPDB dataset are able to address this limitation. Especially, the phrase-level and word-level pairwise text classification task has an positive impact on phrase embeddings. Furthermore, when trained on the paraphrase identification task, the model is able to produce word embeddings that are almost as good as pre-trained word vectors. By combining the characteristics of different training objectives, our model have an advantage in encoding sentences, meanwhile achieving considerable improvement in phrase-level and word-level tasks. We apply different pooling strategies, including mean-pooling, max-pooling and the [CLS] token, and different feature concatenation methods. When investigating the former, we use [u;v;|u−v|] as the input feature for classification, and as for the latter, we choose mean-pooling as the default strategy. Besides, hadamard product u∗v is not helpful in our experiment. Generally, the model is more sensitive to concatenation methods while pooling strategies have a minor influence. For a comprehensive consideration, mean-pooling is preferred in our experiment, and the concatenation of two vectors along with their absolute difference is more suitable than other combinations. In the paraphrase identification task, we randomly select k negative samples for each pair to force the model to identify paraphrases from non-paraphrases. Evidence has shown that the value of k in negative sampling has an impact on phrase and word embeddings (Mikolov et al. When training word embeddings using negative sampling, setting k in the range of 5-20 is recommended for small training datasets, while for large datasets the k can be as small as 2-5. In this ablation experiment, we explore the optimal value of k for our paraphrase identification task. Since the PPDB dataset is extremely large, with more than one million positive pairs, we perform four experiments, in which we only change the value of k and other model settings are maintained the same. Keeping increasing the value of k has no positive effect on phrase and word embeddings and even decreases the performance in sentence tasks.