MOROCO: The Moldavian and Romanian Dialectal Corpus
Table 2: Accuracy rates, weighted F1 scores and macro-averaged F1-scores (in %) for the five evaluation tasks: binary classification by dialect, Moldavian intra-dialect 6-way categorization (by topic), MD→RO cross-dialect 6-way categorization, Romanian (RO) intra-dialect 6-way categorization, and RO→MD cross-dialect 6-way categorization. Results are reported for three baseline models: KRR based on the presence bits string kernel (KRR+k0/16), convolutional neural networks (CNN), and Squeeze-and-Excitation convolutional neural networks (CNN+SE).
['Task', 'Method', 'Validation accuracy', 'Validation weighted  [ITALIC] F1', 'Validation macro  [ITALIC] F1', 'Test accuracy', 'Test weighted  [ITALIC] F1', 'Test macro  [ITALIC] F1']
Results. Regarding the binary classification by dialect task, we notice that all models attain good results, above 90%. SE blocks bring only minor improvements over the baseline CNN. Our deep models, CNN and CNN+SE, attain results around 93%, while the string kernels obtain results above 94%. We thus conclude that written text samples from the Moldavian and the Romanian dialects can be accurately discriminated by both shallow and deep learning models. Regarding the Moldavian intra-dialect 6-way categorization (by topic) task, we notice that string kernels perform quite well in comparison with the CNN and the CNN+SE models. In terms of the macro-averaged F1 scores, SE blocks bring improvements higher than 1% over the baseline CNN. In the MD→RO cross-dialect 6-way categorization task, our models attain the lowest performance on the Romanian test set. We would like to note that in both cross-dialect settings, we use the validation set from the same dialect as the training set, in order to prevent any use of information about the test dialect during training. In other words, the settings are intra-dialect with respect to the validation set and cross-dialect with respect to the test set. The Romanian intra-dialect 6-way categorization task seems to be much more difficult than the Moldavian intra-dialect categorization task, since all models obtain scores that are roughly 20% lower. In terms of the macro-averaged F1 scores, SE blocks bring improvements of around 4% over the baseline CNN. However, the results of CNN+SE are still much under those of the presence bits string kernel. Regarding the RO→MD cross-dialect 6-way categorization task, we find that the models learned on the Romanian training set obtain better results on the Moldavian (cross-dialect) test set than on the Romanian (intra-dialect) test set. Once again, this provides additional evidence that the 6-way categorization by topic task is more difficult for Romanian than for Moldavian. In all the intra-dialect or cross-dialect 6-way categorization tasks, we observe a high performance gap between deep and shallow models. These results are consistent with the recent reports of the VarDial evaluation campaigns Malmasi et al.