Location, Occupation, and Semantics based Socioeconomic Status Inference on Twitter This work was supported by the SoSweet ANR project (ANR-15-CE38-0011) and the ACADEMICS project financed by IDEX LYON.
TABLE II: Classification performance (5-CV): AUC scores (mean ± STD) of three different classifiers on each dataset)
['[EMPTY]', '[BOLD] Census', '[BOLD] Occupation', '[BOLD] Expert']
We hence used it for our predictions in the remainder of this study. We found that the LinkedIn data was the best, with AUC=0.80, to train a model to predict SES of people based on their semantic features. It provided a 10% increase in performance as compared to the census based inference with AUC=0.70, and 19% relative to expert annotated data with AUC=0.61. Thus we can conclude that there seem to be a trade-off between scalability and prediction quality, as while the occupation dataset provided the best results, it seems unlikely to be subject to any upscaling due to the high cost of obtaining a clean dataset. Relying on location to estimate SES seems to be more likely to benefit from such an approach, though at the cost of an increased number of mislabelled users in the dataset. Moreover, the annotator’s estimation of SES using Street View at each home location seems to be hindered by the large variability of urban features. Note that even though inter-agreement is 76%, the Cohen’s kappa score for annotator inter-agreement is low at 0.169. Furthermore, we remark that the expert annotated pipeline was also subject to noise affecting the home location estimations, which potentially contributed to the lowest predictive performance.