Evaluation of Croatian Word Embeddings
Table 2: Detailed results of Croatian word analogy corpus.
['[BOLD] Model', '[BOLD] CBOW', '[BOLD] Skip-gram', '[BOLD] fastText-Skip', '[BOLD] fastText-CBOW']
It seems that free word order and other properties of highly inflected languages from Slavic family have a big impact on the performance of current state-of-the-art word embeddings methods. From results of City-state and City-state-USA category it can be seen that knowledge of the topic in training data has significant impact on performance of a model. We wanted to show differences between two similar categories in case we have an insufficient amount of training data covering a particular topic. Category City-state is showing that model is able to carry such knowledge - if the topic is sufficiently represented in a training data, the model is able to carry this type of information. This behavior is seen in regions from Croatia mentioned in many articles on Croatian Wikipedia, but this was not a case with states from USA. All questions of City-state were covered, but only around 50% of questions in category City-state-USA were in vocabulary. On categories Country-world and EU-cities-states it can be seen that there is no difference between knowledge about states and main cities from EU again state-city pairs from all over the world. Another very poor performance gives group Currency, but this group is usually weak across all languages and shows the weaknesses of the model.