Incremental Active Opinion Learning Over a Stream of Opinionated Documents
Table 1: Requested labels per method and experiment: numbers in percentage regarding the length of the stream including the documents to train the classifier, i.e. the size of the seed. (IG=Information Gain), (U=Uncertainty)
['Experiment + Dataset', 'ACOSTREAM\xa0(IG)', 'ACOSTREAM\xa0(U)', 'IncrementalMNB', 'StaticMNB', 'Random']
In this section, we compare ACOSTREAM using information gain and uncertainty sampling strategies against the IncrementalMNB, the StaticMNB as well as the random sampling based on the performance of kappa over time. This would, however, lead to an unfair comparison as the budget would be spent differently among the strategies. Rather we used different values for the uncertainty threshold α and for random sampling across our experiments yielding to different number of requested labels over the stream. We implemented two experiments: i) we kept the vocabulary fixed over the stream while considering documents that contain only words w∈V, cf. we allow the set of words V to evolve as including new appearing words. ACOSTREAM shows a good performance on both streams when applying information gain sampling. The results expose, upon stream StreamJi, a kappa that is rather close to the kappa of the upper baseline while utilizing only 44% of the true labels (cf. Hence, information gain sampling performs very well requesting only 40% resp. 44% of the labels to achieve a comparable or higher kappa than IncrementalMNB which samples 100% of the labels. In contrast, uncertainty sampling, which uses 40% resp. 47% of the labels on StreamJi resp. TwitterSentiment, shows a lower kappa similarly to the results obtained by random sampling. The reason why kappa drops to 0 at the end of TwitterSentiment is because there only negative documents arrive, consequently one cannot be better than a chance classifier.