All-but-the-Top: Simple and Effective Postprocessing for Word Representations
Table 1: A detailed description for the embeddings in this paper.
['[EMPTY]', '[BOLD] Language', '[BOLD] Corpus', '[BOLD] dim', '[BOLD] vocab size', '[BOLD] avg. ∥ [ITALIC] v( [ITALIC] w)∥2', '∥ [ITALIC] μ∥ [BOLD] 2']
We test our observations on various word representations: four publicly available word representations ( Let v(w)∈Rd be a word representation for a given word w in the vocabulary V. We observe the following two phenomena in each of the word representations listed above: [leftmargin=*,,topsep=0pt] {v(w):w∈V} are not of zero-mean: i.e., all v(w) share a non-zero common vector, v(w)=~v(w)+μ, where μ is the average of all v(w) ’s, i.e., μ=1/|V|∑w∈Vv(w). The norm of μ is approximately 1/6 to 1/2 of the average norm of all v(w) (cf. {~v(w):w∈V} are not isotropic: Let u1,... ,ud be the first to the last components recovered by the principal component analysis (PCA) of {~v(w):w∈V}, and σ1, ...,σd be the corresponding normalized variance ratio. Each ~v(w) can be written as a linear combinations of u: ~v(w)=∑di=1αi(w)ui. This suggests there exists D