AraNet: A Deep Learning Toolkit for Arabic Social Media
Table 3: Model performance in accuracy. UBC_TW_Gender refers to the model trained on UBC Twitter Gender dataset. Gender_Comb denotes the model trained on the Arab-Tweet and UBC-Gender combined TRAIN data split. Each model is evaluated on the corresponding DEV and TEST sets.
['[EMPTY]', '[BOLD] DEV', '[BOLD] TEST']
UBC Twitter Gender Dataset. We also develop an in-house Twitter dataset for gender. We manually labeled 1,989 users from each of the 21 Arab countries. The data had 1,246 “male”, 528 “female”, and 215 unknown users. We remove the “unknown” category and balance the dataset to have 528 from each of the two “male” and “female” categories. We ended with 69,509 tweets for “male” and 67,511 tweets for “female”. We split the users into 80% TRAIN (110,750 tweets for 845 users), 10% DEV (14,158 tweets for 106 users), and 10% TEST (12,112 tweets for 105 users). We then model this dataset with BERT and evaluate on DEV and TEST. on DEV and 60.54% acc. on TEST. These results are 2.89% and 4.76% less than performance on Arab-Tweet, perhaps reflecting more diversity in UBC-Gender data which also makes it more challenging. Another potential reason for this accuracy drop could be that, for this tweet-level task, some tweets from the same user occur across our TRAIN/DEV/TEST splits. This was unavoidable since Arab-Tweet is distributed without user ids, thus not making it possible for us to prevent user-level data leakage into the two tweet-level classification tasks of age and gender we report here. We alleviate this issue for gender by annotating and developing on UBC-Gender where we control for user-level data distribution across the splits as explained earlier.