Durkheim Project Data Analysis Report
Table 15: Mutual Information Cuts
['[EMPTY]', 'Mean Accuracy', 'Std. Dev']
Perhaps it is a mistake, then to cut rare words and rare word-pairs, when using a bag-of-pairs model? In this experiment, no rare words or pairs were cut; only the cut for the MI was altered. Thus, somehow, when word-pairs come into play, failing to cut rare words and phrases makes things worse!