Simple and Effective Paraphrastic Similarity from Parallel Translations
Table 5: Spearman’s ρ×100 between average performance on the 2012-2016 STS tasks compared to SP overlap (SP Ovl.) and language distance as defined by Littell et al. (2017). We included correlations for all languages as well as those with low and high SP overlap with English.
['Model', 'SP Ovl.', 'Lang. Distance']
There are also two clusters in the plot, languages that have a similar alphabet to English and those that do not. In each cluster we find that performance is negatively correlated with language distance. Therefore, languages similar to English yield better performance. When choosing a language to pair up with English for learning paraphrastic embeddings, ideally there will be a lot of SP overlap. However, beyond or below a certain threshold (approximately 0.3 judging by the plot), the linguistic distance to English is more predictive of performance. Of the factors in URIEL, syntactic distance was the feature most correlated with STS performance in the two clusters with correlations of -56.1 and -29.0 for the low and high overlap clusters respectively. This indicates that languages with similar syntax to English helped performance. One hypothesis to explain this relationship is that translation quality is higher for related languages, especially if the languages have the same syntax, resulting in a cleaner training signal.