AraNet: A Deep Learning Toolkit for Arabic Social Media
Table 4: Distribution of classes within the MADAR twitter corpus.
['[EMPTY]', '[BOLD] # of tweets  [BOLD] TRAIN', '[BOLD] # of tweets  [BOLD] DEV', '[BOLD] # of tweets  [BOLD] TEST']
The dialect identification model in AraNet is based on our winning system in the MADAR shared task 2 [bouamor-etal-2019-madar] as described in [zhang2019no]. The corpus is divided into training, development, and test; and the organizers masked test set labels. We lost some tweets from TRAIN when we crawled using tweet ids, ultimately acquiring 2,036 (TRAIN-A), 281 (DEV) and 466 (TEST). We also make use of the task 1 corpus (95,000 sentences [bouamor2018madar]). More specifically, we concatenate the task 1 data to the training data of task 2, to create TRAIN-B. Again, note that TEST labels were only released to participants after the official task evaluation. More information about the data is in [bouamor2018madar]. We use TRAIN-A to perform supervised modeling with BERT and TRAIN-B for self training, under various conditions. We refer the reader to [zhang2019no] for more information about our different experimental settings on dialect id. We acquire our best results with self-training, with a classification accuracy of 49.39% and F_{1} score at 35.44. This is the winning system model in the MADAR shared task and we showed in [zhang2019no] that our tweet-level predictions can be ported to user-level prediction. On user-level detection, our models perform superbly, with 77.40% acc. and 71.70% F_{1} score on unseen MADAR TEST.