DiaNet: BERT and Hierarchical Attention Multi-Task Learning of Fine-Grained Dialect
Table 3: Results on TEST with models exploiting noisy labels on 20 countries (with Djibouti excluded). For comparison, our gold (BERT trained on human-labeled TRAIN) is re-trained with 20 classes.
['[BOLD] Supervision', '[BOLD] acc', '[BOLD] F1']
We only perform experiments on predicting country labels. Our goal is to answer the question “To what extent can automatically acquired labels in our dataset be beneficial for learning?”. To this end, we remove human annotated users from our larger automatically labeled pool and use only data tagged with any of the 319 cities whose users we manually labeled. Keeping tweets with at least 3 Arabic words, we acquire 1,161,651 tweets from 3,195 users, across all countries except Djibouti (all whose users were already in our human annotation round). As such, we have 20 countries in this dataset and refer to it simply as Auto-Tagged. We exploit Auto-Tagged in 3 experimental settings, reporting results on our gold TEST in all 3 cases. The 3 settings are: (1) Weakly Supervised: Where fine-tune BERT exclusively on Auto-Tagged; (2) Weak+Gold: Where concatenate Auto-Tagged and our TRAIN (gold), shffle the dataset, and fine-tune BERT on it; and (3) Weak-Then-Gold: Where fine-tune BERT on Auto-Tagged first, then resume fine-tuning on our human labeled data (TRAIN).