Universal Sentence Encoder
Table 2: Model performance on transfer tasks. USE_T is the universal sentence encoder (USE) using Transformer. USE_D is the universal encoder DAN model. Models tagged with w2v w.e. make use of pre-training word2vec skip-gram embeddings for the transfer task model, while models tagged with lrn w.e. use randomly initialized word embeddings that are learned only on the transfer task data. Accuracy is reported for all evaluations except STS Bench where we report the Pearson correlation of the similarity scores with human judgments. Pairwise similarity scores are computed directly using the sentence embeddings from the universal sentence encoder as in Eq. (1).
['Model', 'MR', 'CR', 'SUBJ', 'MPQA', 'TREC', 'SST', 'STS Bench']
We observe that transfer learning from the transformer based sentence encoder usually performs as good or better than transfer learning from the DAN encoder. Hoewver, transfer learning using the simpler and fast DAN encoder can for some tasks perform as well or better than the more sophisticated transformer encoder. Models that make use of sentence level transfer learning tend to perform better than models that only use word level transfer. The best performance on most tasks is obtained by models that make use of both sentence and word level transfer.