Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings
Table 4: Results on SemEval textual similarity datasets (Pearson’s r×100). The highest score in each row is in boldface.
['Dataset', 'LSTMavg', 'avg', 'GRAN']
The LSTMavg and gated recurrent averaging network are more closely correlated in performance, in terms of Spearman’s ρ and Pearson’r r, than either is to avg. But they do differ significantly in some datasets, most notably in those comparing machine translation output with its reference. Interestingly, both the LSTMavg and gated recurrent averaging network significantly outperform avg in the datasets focused on comparing glosses like OnWN and FNWN. Upon examination, we found that these datasets, especially 2013 OnWN, contain examples of low similarity with high word overlap. For example, the pair ⟨the act of preserving or protecting something., the act of decreasing or reducing something.⟩ from 2013 OnWN has a gold similarity score of 0.4. It appears that avg was fooled by the high amount of word overlap in such pairs, while the other two models were better able to recognize the semantic differences.