Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English
Table 1. Percent of the 2,500 tweets in each bin classified as English by each classifier; Diff. is the difference (disparity on an absolute scale) between the classifier accuracy on the AA-aligned and white-aligned samples. t is the message length for the bin.
['[ITALIC] langid.py', '[ITALIC] t≤5', 'AA Acc. 68.0', 'WH Acc. 70.8', 'Diff. 2.8']
From manual inspection, we observed that longer tweets are significantly more likely to be correctly classified, which is a potential confound for a race disparity analysis, since the length distribution is different for each demographic group. For each bin, we sampled 2,500 AA-aligned tweets and 2,500 white-aligned tweets, yielding a total of 20,000 messages across the two categories and four bins. We limited pre-processing of the messages to fixing of HTML escape characters and removal of URLs, keeping “noisy” features of social media text such as @-mentions, emojis, and hashtags. We then calculated, for each bin in each category, the number of messages predicted to be in English by each classifier. Accuracy results are shown in Table