Evaluation of Sentence Embedding Models for Natural Language Understanding Problems in Russian
Table 4: FastText embeddings results.
['Method', '[BOLD] MCQA Accuracy', '[BOLD] MCQA F1', '[BOLD] NSP Accuracy', '[BOLD] NSP F1', '[BOLD] PI Accuracy', '[BOLD] PI F1']
The best quality on each task was reached by CatBoost method according to Accuracy metrics. According to the F1 score, CatBoost also outperforms other methods in all of the tasks except Paraphrase identification where the best score with a large margin was achieved by the unsupervised method. There can also be seen a big quality difference between CatBoost and other methods in next sentence prediction task. Logistic regression shows the worst results in each task. The overall performance of methods based on FastText model is far from perfect. The results achieved using FastText can be considered as a baseline for our investigation.