Nowcasting the Stance of Social Media Users in a Sudden Vote: The Case of the Greek Referendum
Table 2. Number of users (u) and tweets (t) used in our experiments per evaluation day.
['day', '[BOLD] 0', '[BOLD] 1', '[BOLD] 2', '[BOLD] 3', '[BOLD] 4', '[BOLD] 5', '[BOLD] 6', '[BOLD] 7', '[BOLD] 8']
Two authors of the paper (Greek native speakers) annotated each of the users in the test set, using the tweets after the referendum announcement. Each annotator was allowed to label an account as YES, NO, or N /A, if uncertain. To mimic a real-time scenario, we refrained from balancing our train/test sets, since it would have been rather impossible to know the voting intention distribution of Twitter users a-priori. We train and test our models based on the data collected on a daily basis (every midnight), starting from the day before the announcement of the referendum (day 0) until the day before its due date (day 8). This way, we mimic a real-time setting and gain better evaluation insights. To evaluate our models, we compute the macro-average F-score, which forms a more challenging metric compared to micro-averaging, given the imbalanced distribution of our test set. At each evaluation time point t, we use information about the users in our training set up to t, to classify the test users that have tweeted at least once up to t (note that all of the users in our training set have tweeted before the announcement of the referendum, thus the size of the training set in terms of number of users remains constant). However, we did not observe any major differences in our evaluation by excluding newly added users. Parameter selection is performed on every evaluation day using a 5-fold cross-validation on the training set.