Parallelizing Word2Vec in Shared and Distributed Memory
TABLE IV: Predictive performances of our distributed word2vec trained on the one billion word benchmark [13] evaluated on the word similarity and word analogy tasks. The performance of the original word2vec is provided as baseline.
['#Nodes Original ( [ITALIC] N=1)', 'Word Similarity 64.0', 'Word Similarity 64.0', 'Word Analogy 32.4', 'Word Analogy 32.4']
Next we demonstrate the scalability and predictive performance of our distributed word2vec on multi-node distributed systems. The experiments with our distributed word2vec are performed on two CPU clusters: (1) Intel Broadwell nodes connected via FDR Infiniband, and (2) Intel KNL nodes connected via Intel OPA Fabric. Fig. For the purpose of comparison, we also include in Fig. Again, good scalability is only meaningful when similar or better accuracy is achieved. We therefore provide the predictive performance of the original word2vec As can been seen from Fig. As the number of nodes increases, to achieve the linear scaling while maintaining a comparable accuracy, we need to increase the learning rate and the model synchronization frequency slightly to mitigate the loss of convergence rate. When number of Broadwell nodes increases to 32 (or KNL for 16), we need to further increase model synchronization frequency to maintain a good predictive accuracy. However, the increment of model synchronization frequency takes a toll on the scalability, and leads to a sub-linear scaling at 32 Broadwell nodes or 16 KNL nodes. Despite of this, our distributed word2vec delivers over 100 million words/sec with a small 1% accuracy loss. To the best of our knowledge, this is the best performance reported so far on this benchmark. As a comparison, BIDMach only delivers 2.4X speedup on 4 GPU cards vs. 1 GPU card or a 60% efficiency.