A Robust Framework for Classifying Evolving Document Streams in an Expert-Machine-Crowd Setting
TABLE III: Classification accuracies in terms of AUC for all the datasets before and after the data improvements process (Results of the Sympathy and support class are omitted due to space limitations).
['Categories  [ITALIC] Ti Dataset', 'Categories  [ITALIC] Ti Classifier', 'Affected individuals Before', 'Affected individuals After', 'Caution and advice Before', 'Caution and advice After', 'Donations and volunteering Before', 'Donations and volunteering After', 'Infrastructure and utilities Before', 'Infrastructure and utilities After']
Two training sets are formed (i.e. original labels and labels after the data improvements process). We remove stop-words, URLs, and user mentions from the items. Stemming is performed using the Lovins stemmer. Uni-grams and bi-grams are used as features and we used the information gain feature selection method to select top 1k features. We use three well-known learning algorithms: SVM (support vector machines), Naive Bayes (NB), and Random forest (RF). We perform the evaluation of the learned model using 10-fold cross validation. A substantial gain in AUC can be clearly observed in case of the improved training data.