A Robust Self-Learning Method for Fully Unsupervised Cross-Lingual Mappings of Word Embeddings: Making the Method Robustly Reproducible as Well
Table 2: Ablation study of the algorithm proposed by Artetxe et al., (2018b) on the same four languages. We performed seven ablations and report the original and reproduced best accuracy (best), average accuracy (avg), success rate (s) and average runtime in minutes (t). Bold values represent significant differences between the original and reproduced results. *We did not reproduce this ablation due to time constraints as more than three days per run would have been required to reach convergence. †The authors reported a bug on the code they used for generating the CSLS values which made their code only yield zero accuracies.
['[EMPTY]', '[BOLD] EN-DE best', '[BOLD] EN-DE avg', '[BOLD] EN-DE s', '[BOLD] EN-DE t', '[BOLD] EN-ES best', '[BOLD] EN-ES avg', '[BOLD] EN-ES s', '[BOLD] EN-ES t', '[BOLD] EN-FI best', '[BOLD] EN-FI avg', '[BOLD] EN-FI s', '[BOLD] EN-FI t', '[BOLD] EN-IT best', '[BOLD] EN-IT avg', '[BOLD] EN-IT s', '[BOLD] EN-IT t']
Amongst other things, we note that the accuracy results we obtained are all within the 95 % confidence interval given by our 25 runs, with the only exceptions being the unsupervised initialization ablation and the runtimes. Our results also showed great differences in the algorithm ’s runtimes, even though we used the same graphics card as the original paper. We also point out that not all ablation study configurations can be run with the same compute power, i.e. when removing the vocabulary cutoff parameter, the matrices no longer fit inside a GPU’s memory and we have to prepare additional RAM space and CPU resources to run the script. However, when we attempted to reproduce the frequency-based vocabulary cutoff ablation where n is set to 100k, we were unable to obtain a single run to complete even after 3 days of computations.