Pointwise Paraphrase Appraisal is Potentially Problematic
Table 4: Percentage of paraphrase and non-paraphrase pairs with higher paraphrase score (%) than a pair of identical sentences given the same query sentence. Avg Score Differences: average score difference between paraphrase/non-paraphrase and identical pairs. (Only pairs with higher scores than the identical ones are included.)
['[EMPTY]', 'QQP → QQP', 'QQP → PAWSQQP', 'QQP + PAWSQQP', 'MRPC', 'Twitter URL']
We then compared each sentence pair, (s,s′), to a pair of identical sentences, (s,s), given the same query sentence s. We fine-tuned BERT on each dataset to learn a paraphrase score function f, and computed the fraction of tests where a randomly-selected pair gets a higher paraphrase score than an identical pair, f(s,s′)>f(s,s). The model trained on QQP considers more than 30% of randomly-selected paraphrase sentence pairs to be more similar than the identical pairs, but the ratio decreases to 21% when adding the adversarial set into training. For MRPC and Twitter URL corpus, less than 5% of paraphrase pairs are considered to be more similar than the identical pairs. For a randomly-selected sentence pair, (s,s′), and a pair of identical sentences, (s,s), given the same query s sentence, we computed the score difference as f(s,s′)−f(s,s). After augmenting the training set with PAWSQQP, the right tail of the distribution for paraphrase pairs diminishes. This indicates that the model considers fewer non-identical sentences as more similar to the query sentence than itself.