Are All Good Word Vector Spaces Isomorphic?
Table 3: Reference BLI (MRR reported) and isomorphism scores (all three measures discussed in the main paper are reported, computed on L2-normalised vectors) in a setting where we fully train both English and Spanish monolingual vector spaces (i.e., training lasts for 15 epochs for both languages) on the full data, without taking snapshots at earlier stages, and without data reduction simulation experiments.
['[BOLD] Full EN Vector Space – Full ES Vector Space', '[BOLD] Full EN Vector Space – Full ES Vector Space HFREQ', '[BOLD] Full EN Vector Space – Full ES Vector Space MFREQ', '[BOLD] Full EN Vector Space – Full ES Vector Space LFREQ', '[BOLD] Full EN Vector Space – Full ES Vector Space PANLEX', '[BOLD] Full EN Vector Space – Full ES Vector Space MUSE']
It provides “reference” BLI scores and scores stemming from isomorphism measures when we align fully trained EN and ES spaces, that is, when we rely on standard 15 epochs of fastText training on respective Wikipedias. In other words, one vector space is fully trained, while we take early-training snapshots (after seeing only 10M, 20M, …, 100M word tokens in training) of the other vector space. The results again stress the importance of training corpus size as well as training duration—early training stages clearly lead to suboptimal performance and non-isomorphic spaces. However, such shorter training durations (in terms of the number of tokens) are often encountered “in the wild” with low-resource languages. The results with 1k seed translation pairs are provided in the main paper. Figure versus L2 + mean centering + L2) on the RSIM isomorphism scores in different training regimes for en-ar and en-ja experiments. It provides additional isomorphism scores, not reported in the paper, again stressing the importance of monolingual vector space preprocessing before learning any cross-lingual mapping.