Learning Better Universal Representations from Pre-trained Contextualized Language Models
Table 1: Examples from the PPDB database. Pairs of phrases and words are annotated with similarities computed from the Google n-grams and the Annotated Gigaword corpus, entailment labels and the score for each label.
['Target', 'Paraphrase', 'GoogleNgramSim', 'AGigaSim', '[ITALIC] equivalence score', 'Entailment label']
Pairwise Text Classification Apart from the paraphrase identification task, we design a phrase-level and word-level pairwise text classification task to make use of the phrasal entailment information in the PPDB dataset. For each paraphrase pair (A,B), the model is trained to recognize from three types of relationships: equivalence, entailment and independent. This task is more challenging than the previous one, because the model tries to capture the degree of similarity between phrases and words while words are considered dissimilar even if they are closely related. A one-layer classifier is used to determine the entailment label c for each pair: P(c|A,B)=softmax(W⊤3⋅[u;v;|u−v|])