1 Introduction
Table 3: Average difference (absolute percentage) to the oracle downstream instability when using embedding distance measures as the selection criteria for dimension and precision parameters under fixed memory budgets. Top-performing values are bolded.
['[ITALIC] Downstream Task  [ITALIC] Embedding Algorithm', 'SST-2 CBOW', 'SST-2 GloVe', 'SST-2 MC', 'Subj CBOW', 'Subj GloVe', 'Subj MC', 'CoNLL-2003 CBOW', 'CoNLL-2003 GloVe', 'CoNLL-2003 MC']
For the second, more challenging setting, we enumerate all embedding pairs with different dimension-precision combinations which correspond to the same total memory. For each embedding measure, we report the average absolute percentage difference between the downstream instability of the pair selected by the measure to the most stable “oracle” pair, across different memory budgets. We also introduce two naive baselines that do not require an embedding distance measure: high precision, which selects the pair with the highest precision possible at each memory budget, and low precision, which selects the pair with the lowest precision possible at each memory budget. As before, we repeat over three seeds, comparing embedding pairs of the same seed, and report the average. We see that the eigenspace instability measure and k-NN measure again outperform the other baselines on the majority of downstream tasks, with the eigenspace instability measure attaining a distance up to 2.98% (absolute) closer to the oracle than the other baselines, and average distance to the oracle 0.03% (absolute) better to 0.35% (absolute) worse than the k-NN measure across downstream tasks