Towards Quantifying the Distance between Opinions
Table 6: Performance comparison of the distance measures on all 18 datasets. The semantic distance in opinion distance (OD) measure is computed via cosine distance over either Word2vec (OD-w2v with semantic distance threshold 0.6) or Doc2vec (OD-d2v with distance threshold 0.3) embeddings. Sil. refers to Silhouette Coefficient. The second best result is italicized and underlined. The ARI and Silhouette coefficients scores of both OD methods (OD-d2v and OD-w2v) are statistically significant (paired t-test) with respect to baselines at significance level 0.005.
['Topic Name', 'Size', 'TF-IDF ARI', 'WMD ARI', 'Sent2vec ARI', 'Doc2vec ARI', 'BERT ARI', '[ITALIC] OD-w2v ARI', '[ITALIC] OD-d2v ARI', 'TF-IDF  [ITALIC] Sil.', 'WMD  [ITALIC] Sil.', 'Sent2vec  [ITALIC] Sil.', 'Doc2vec  [ITALIC] Sil.', 'BERT  [ITALIC] Sil.', '[ITALIC] OD-w2v  [ITALIC] Sil.', '[ITALIC] OD-d2v  [ITALIC] Sil.']
The semantic threshold for OD-d2v is set at 0.3 while for OD-w2v is set at 0.6. [CONTINUE] We evaluate our distance measures in the unsupervised setting, specifically, evaluating the clustering quality using the Adjusted Rand Index (ARI) and Silhouette coefficient. We benchmark against the following baselines: WMD (which relies on word2vec embeddings), Doc2vec and TF-IDF. The results are shown in Table 6. The ARI and Silhouette coefficients scores of both OD methods (OD-d2v and OD-w2v) are statistically significant (paired t-test) with respect to baselines at significance level 0.005. [CONTINUE] Opinion distance methods generally outperform the competition on both ARI and Silhouette coefficient. [CONTINUE] This is reflected in the average ARI and average Silhouette coefficients of the baseline distance measures. [CONTINUE] In the exceptional case of "Hydroelectric Dams" dataset, the opinion distance OD performs particularly bad compared to TF-IDF