1 Introduction
Table 1: Spearman correlation scores between embedding distance measures and downstream prediction disagreement across varying dimension-precision pairs for the embedding. Downstream models are trained for sentiment analysis (SST-2, Subj) and NER (CoNLL-2003) tasks. Strongest correlation values are bolded.
['[ITALIC] Downstream Task  [ITALIC] Embedding Algorithm', 'SST-2 CBOW', 'SST-2 GloVe', 'SST-2 MC', 'Subj CBOW', 'Subj GloVe', 'Subj MC', 'CoNLL-2003 CBOW', 'CoNLL-2003 GloVe', 'CoNLL-2003 MC']
We evaluate how predictive the eigenspace instability measure is of downstream instability, showing that the theoretically grounded eigenspace instability measure correlates strongly with downstream instability and is competitive with other embedding distance measures. To do this, we measure the Spearman correlations between the downstream prediction disagreement and the embedding distance measure for each of the five tasks and three embedding algorithms. The Spearman correlation quantifies how similar the ranking of the pairs of embeddings based on the embedding distance measure is to the ranking of the pairs of embeddings based on their downstream prediction disagreement, with a maximum value of 1.0. Moreover, the strong correlation of at least 0.68 for the eigenspace instability measure across embedding algorithms and downstream tasks validates our theoretical claim that this measure relates to downstream disagreement. We now provide additional results validating the strong relationship between the eigenspace instability measure and downstream instability. We see that k-NN measure and the eigenspace instability measure achieve strong correlations since the lines are generally monotonically increasing for both CBOW and MC embedding algorithms.