PAWS: Paraphrase Adversaries from Word Scrambling
Table 7: Accuracy (%) of classification and AUC scores (%) of precision-recall curves on Quora Question Pairs (QQP) testing set and our PAWSQQP development set. QQP→PAWSQQP indicates that models are trained on QQP and evaluated on PAWSQQP. Other columns are defined in a similar way. QQP+PAWSQQP is a simple concatenation of the two training sets. Boldface numbers indicate the best accuracy for each testing scenario. Numbers in parentheses indicate absolute gains from adding PAWSQQP training data.
['Models', 'QQP→QQP (Acc)', 'QQP→QQP (AUC)', 'QQP→PAWSQQP (Acc)', 'QQP→PAWSQQP (AUC)', 'QQP+PAWSQQP→PAWSQQP (Acc)', 'QQP+PAWSQQP→PAWSQQP (AUC)']
We first train models on the Quora Question Pairs (QQP) training set, and column “QQP→QQP” shows that all models achieve over 83% accuracy on QQP. However, when evaluating on PAWSQQP, all models, including BERT, obtain abysmal accuracy under 40% (column “QQP→PAWSQQP”). We hypothesize the performance on PAWSQQP relies on two factors: the number of representative training examples, and the capability of models to represent complex interactions between words in each sentence and across the sentences in the pair. As expected, all models benefit from new training examples, but to different extents. Gains are much larger on state-of-the-art models like BERT, while the BOW model learns almost nothing from new examples. As a consequence, performance changes are more drastic on PAWSQQP than on QQP. For example, the absolute difference between BiLSTM and BERT is 4.2% on QQP, but it goes up to 27% on PAWSQQP, which is a 60% relative reduction in error.