Pointwise Paraphrase Appraisal is Potentially Problematic
Table 3: The percentage (%) of sentence pairs with asymmetrical prediction results. Reverse Order: sentence pairs predicted with different labels when reversing the order of the sentences. Identical: identical pairs that are predicted as non-paraphrases. (Please see Section 2.3 for actual data sizes.)
['Datasets', 'Reverse Order BERT', 'Reverse Order BOW', 'Identical BERT', 'Identical BOW']
In the original datasets, each sentence pair is only concatenated in one way as (s1,s2) and a label y will be predicted by the model. We constructed new sentence pairs in the reverse order as (s2,s1), and tested the model on these sentence pairs and got their predicted labels y′. To find out how much it would affect the prediction results, we computed the ratio of sentence pairs that are predicted with a different label (y≠y′). We would like to know if the asymmetrical structure also affect BERT’s ability to identify identical sentences as paraphrases. We collected distinct sentences for each dataset and constructed a new set of sentence pairs by pairing each one with itself. Each pair is labeled as paraphrase. We calculated the ratio of pairs that are predicted as non-paraphrase by the model. recognizes 2.4% of identical pairs as non-paraphrases and the ratio increases about 5% when tested on PAWSQQP. BOW trivially achieves perfect accuracy on pairs of identical sentences, since they have exactly the same bags of words. Selection Bias in the Pointwise Setting. Datasets tend to have inconsistent frequency of sentences causing the model biased towards the dominating sentences. For instance, we found that some sentences from the Twitter corpus are repeated almost a hundred times as the first input sentence.