Improving zero-shot learning by mitigating the hubness problem
Figure 2: Percentage accuracy scores for En→It translation. Left: No regularization and Generalized Cross-validation (GCV) regression varying the training size. Right: Results split by frequency bins, 5K training and no regularization.
['Train size', 'No regularization NN', 'No regularization NNnrm', 'No regularization GC']
We report results without regularization as well as with the regularization parameter λ estimated by generalized cross-validation (GCV) (Hastie Both corrected methods achieve significant improvements over standard NN, ranging from 7% to 14%. For the standard method, the performance decreases as the training data size increases beyond 5K, probably due to the noise added by lower-frequency words. The corrected measures are robust against this effect: adding more training data does not help, but it does not harm them either. Regularization does not improve, and actually hampers the standard method, whereas it benefits the corrected measures when using a small amount of training data (1K), and does not affect performance otherwise. The results by frequency bin show that most of the improvements are brought about for the all-important medium- and low-frequency words. Although not directly comparable, the absolute numbers we obtain are in the range of those reported by Mikolov et al. Furthermore, we observe, similarly to them, that the accuracy scores underestimate the actual performance, as many translations are in fact correct but not present in our gold dictionary.