On the Ground Validation of Online Diagnosis with Twitter and Medical Records
Table 1: Probability of keywords being Tweeted by a user during the month that he or she was diagnosed with influenza.
['Word', 'Total', 'Odds Ratio', 'Significance']
First, we use the occurrence or absence of keywords as features for classification. A set of keywords are defined that are possibly signals of influenza. We chose {flu, influenza, sick, cough, cold, medicine, fever} as our set of keywords. These keywords include the names and symptoms of the illness in addition to “medicine” and serve as a set of keywords that may have been chosen by a domain expert. Additionally, we try algorithmically selecting keywords by first finding the 12,393 most common keywords in the data set. We then rank them based off of information gain on predicting influenza and choose the top 10, 100 or 1000 keywords from the list. We use Naive Bayes, random forest, J48 (a Java implementation of C4.5), logistic regression and support vector machines to classify a user as being sick in a given month or not