1 Introduction
Table 10: Worst-case absolute percentage error when using each embedding distance measure to predict the most stable embedding parameters on downstream tasks over of all pairs of parameters. Downstream models are trained for sentiment (SST-2, Subj) and NER (CoNLL-2003) tasks. Lowest errors are bolded.
['[ITALIC] Downstream Task  [ITALIC] Embedding Algorithm', 'SST-2 CBOW', 'SST-2 GloVe', 'SST-2 MC', 'Subj CBOW', 'Subj GloVe', 'Subj MC', 'CoNLL-2003 CBOW', 'CoNLL-2003 GloVe', 'CoNLL-2003 MC']
We also evaluate the the worst-case performance of the embedding distance measures when used as a selection criterion for dimension-precision parameters. First, on the easier task of choosing the more stable dimension-precision pair out of two choices, we define the worst-case performance as the maximum increase in instability that may occur by using the embedding distance measure to choose the dimension-precision parameters (rather than the ground truth choice). On the more challenging task of choosing the most stable dimension-precision pair under a memory budget, we define the worst-case performance as the worst-case absolute percentage error to the oracle parameters under a given memory budget.