Introducing two Vietnamese Datasets for Evaluating Semantic Models of (Dis-)Similarity and Relatedness
Bang 1: Inter-annotator agreements measured by Spearman’s ρ, Krippendorff’s α, and the average standard deviation (STD) of all pairs across word classes.
['[EMPTY]', '[ITALIC]  [BOLD] All', '[BOLD] Noun', '[BOLD] Verb', '[BOLD] Adjective']
The overall IAA-pairwise of ViSim-400 is ρ=0.79, comparing favourably with the agreement on the SimLex-999 dataset ( ρ=0.67 using the same IAA-pairwise measure). Regarding IAA-mean, ViSim-400 also achieves an overall agreement of ρ=0.86, which is similar to the agreement in \newciteVulic:16, ρ=0.86. For Krippendorff’s α coefficient, the value achieves α=0.78, also reflecting the reliability of the annotated dataset.