Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings
Table 7: Illustrative sentence pairs from the STS datasets showing errors made by LSTMavg and avg. The last three columns show the gold similarity score, the similarity score of LSTMavg, and the similarity score of avg. Boldface indicates smaller error compared to gold scores.
['#', 'Sentence 1', 'Sentence 2', 'Lavg', 'avg', 'Gold']
We analyze the predictions of avg and the recurrent networks, represented by LSTMavg, on the 20 STS datasets. We choose LSTMavg as it correlates slightly less strongly with avg than the GRAN on the results over all SemEval datasets used for evaluation. We scale the models’ cosine similarities to lie within [0,5], then compare the predicted similarities of LSTMavg and avg to the gold similarities. We analyzed instances in which each model would tend to overestimate or underestimate the gold similarity relative to the other. We find that avg tends to overestimate the semantic similarity of a sentence pair, relative to LSTMavg, when the two sentences have a lot of word or synonym overlap, but have either important differences in key semantic roles or where one sentence has significantly more content than the other. Conversely, avg tends to underestimate similarity when there are one-word-to-multiword paraphrases between the two sentences as shown in examples 3 and 4.