Word Embeddings and Stability
Table 2.4: Estimations of the parameters of the distribution Ψ(T,C,wt,ws) for the target word wt= momentum and different query words ws, sorted by similarity. The estimation is based on 32 runs of word2vec (skip-gram) trained on the English Wikipedia, with a reduced vocabulary size of 200,000 words. The two columns on the right describe the predicted probability for the different query words to appear as the nearest neighbors of the target word based on Equations (2.24) and (A.15).
['target word  [ITALIC] wt=\xa0 [BOLD] momentum  [BOLD] rank', 'target word  [ITALIC] wt=\xa0 [BOLD] momentum  [BOLD] query word', 'target word  [ITALIC] wt=\xa0 [BOLD] momentum  [BOLD] mean  [ITALIC] μ', 'target word  [ITALIC] wt=\xa0 [BOLD] momentum  [BOLD] std.  [ITALIC] σ', 'target word  [ITALIC] wt=\xa0 [BOLD] momentum  [ITALIC] p\\#1( [ITALIC] wt, [ITALIC] ws)', 'target word  [ITALIC] wt=\xa0 [BOLD] momentum  [ITALIC] p\\#2( [ITALIC] wt, [ITALIC] ws)']
In practice, we can estimate these parameters by sampling a set of embedding spaces {Vi for i=1,... ,r} from the distribution Ω(T,C) , i.e. applying the same embedding technique T to the corpus C subsequently for r times. Then, we measure cos(wt,ws)i for each run and finally use the formulas below to obtain the maximum-likelihood estimation of the parameters of the underlying Normal distribution: μts=1rr∑i=1cos(wt,ws)iσts= ⎷1rr∑i=1[cos(wt,ws)i−μts]2 (2.17) We look at two words: Firstly, the – on average – nearest neighbor w\#1= inertia of the target word wt=momentum and secondly, the word with the 100th largest mean cosine similarity, w\#100 =inelastic. Now we want to estimate p\#1(wt,w\#100), i.e. the probability that a word which usually ranks around position 100, ends up on rank 1 for one run. We can get an upper bound for this probability, by evaluating a weaker condition – the probability that cos(wt,w#100) is larger than cos(wt,w#1). ≈2.74×10−33 (A.2) withA=0.489−0.650√2(0.0092+0.0102) ≈−8.46 (A.3) And this is an upper bound on the probability p\#1(wt,w\#100), thus: p\#1(wt,w\#100)≤2.74×10−33 (A.4) However, this leads to a higher stability than we generally observe in practice.