Finding Eyewitness Tweets During Crises
Table 4: ROC AUC scores for the individual feature classes and top performing feature class combinations trained with differing levels of class imbalance. Comparison to roller2012supervised.
['[BOLD] Individual Feature Classes', '[BOLD] Boston Bombing', '[BOLD] Hurricane Sandy']
To understand how our feature classes perform under differing levels of class imbalance, we repeat the experiments from the previous two sections, varying the class imbalance from all-positive (all-IR) to all-negative (all-OR) instances. We calculate the receiver operating characteristic (ROC) area under the curve (AUC) score for each individual feature class and for each of the top-performing combinations of feature classes. Our combinations outperform our baseline of the \newciteroller2012supervised model by 9.2% in the Boston Bombing and 31.8%. One interesting result is that while combinations seem ideal in the case of the 50/50 split, the bigram feature set achieves the best results in the case of Hurricane Sandy.