Detecting Perceived Emotions in Hurricane Disasters
Table 11: Pre-training using 50-1600K unlabeled samples from Sentiment. See Table 6 for styling considerations.
['[EMPTY]', 'AGR', 'OPT', 'LOV', 'SBM', 'AWE', 'DSP', 'RMR', 'CNT', 'AVG']
The Sentiment and HurricaneExt datasets contain significantly more samples than currently used. In this section, we study the effects of using varying amounts of pre-training data on downstream HurricaneEmo performance. For both pre-training datasets, we use 1.6M samples. For both types of pre-training tasks, there is no noticeable benefit to using more pre-training data. The supervised Sentiment and unsupervised HurricaneExt results both saturate around 200K samples, which is what we report in our paper. The results for unsupervised HurricaneExt pre-training are especially compelling because they show that, without any labeled data, we can achieve strong downstream results. Finally, the unsupervised Sentiment task yields almost no gains for most emotions, showing that the type of data used for masked language modeling matters. Through side-by-side comparisons, we notice that the Sentiment samples are shorter in length and the HurricaneExt samples contain more relevant content, such as hurricane-specific hashtags.