1 Introduction
Table 2: Selection error when using embedding distance measures to predict the most stable embedding dimension-precision parameters on sentiment analysis (SST-2, Subj) and NER (CoNLL-2003) downstream tasks. Lowest errors are bolded.
['[ITALIC] Downstream Task  [ITALIC] Embedding Algorithm', 'SST-2 CBOW', 'SST-2 GloVe', 'SST-2 MC', 'Subj CBOW', 'Subj GloVe', 'Subj MC', 'CoNLL-2003 CBOW', 'CoNLL-2003 GloVe', 'CoNLL-2003 MC']
For the first, simpler setting, we first form all groupings of two embedding pairs with different dimension-precision combinations. For instance, a grouping may have one embedding pair with dimension 800, precision 32, and another embedding pair with dimension 200, precision 2, where a pair consists of a Wiki’17 and a Wiki’18 embedding from the same algorithm. For each embedding distance measure, we report the fraction of groupings where the embedding distance measure correctly chooses the embedding pair with lower downstream instability on a given task. We repeat over three seeds, comparing embedding pairs of the same seed, and report the average. Moreover, across downstream tasks, the eigenspace instability measure attains 0.95× to 1.55× the error rate of the k-NN measure.