neuralVietNLP: neural NLP Tools for Vietnamese Language Processing
Bang 1: Evaluation results of different word embeddings on the Word Analogy Task. P-value column shows significance test results using Paired t-tests. ‘*’ means significant (p-value ¡ 0.05) to the rest.
['Model', 'MAP@10', 'P-value']
Regarding the evaluation metric, \newciteMikolov:13 used accuracy metric to measure the quality of word embeddings on the task in which only when the expected word is on top of the prediction list, then the model gets +1 for true positive count. However, this is not a well-suited metric in low resource languages where training corpus is relatively small, i.e., 233M tokens in Vietnamese Wiki compared to 6B tokens in Google News corpus. Therefore, we change to use mean average precision (MAP) metric to measure quality of the word analogy task. MAP is widely used in information retrieval to evaluate results based on the topK returned results Vu et al. We use MAP@10 in this paper. The evaluator of ETNLP also shows P-value using the paired t-tests on the raw MAP@10 scores (i.e., before averaging) between different sets of embeddings. The P-values but there is no significant difference between the three. Therefore, these sets of embeddings will be selected for NER task.