Closed Form Word Embedding Alignment
Table 3: RMSE after alignment for embeddings. Left: Created from different datasets. Right: Created by different embeddings; uses AO+Scaling mapping rows onto columns and changing scale.
['→', 'RAW', 'GloVe', 'word2vec']
Dependence of Datasets. The alignment of G(W) and G(WG) has less error than either to G(CC42) and G(CC840), likely because they have substantial overlap in the source data (both draw from Wikipedia). In all cases, the error is roughly on the scale of adding Gaussian noise with σ∈[0.25,0.35] to the embeddings, or reducing the dataset to 10M to 100M tokens. This difference in target and source is significant because the scale inherent in these alignments change, and with it, so does the RMSE. Also as shown, the scale parameter s∗ from GloVe to word2vec in AO+Scaling is approximately 3 (and non-symmetrically about 0.25 in the other direction from word2vec to GloVe). This means for the same alignment, we expect the RMSE to be between 3 to 4 (≈1/0.25) times larger as well.