Durkheim Project Data Analysis Report
Table 12: Bag of Phrases Tuning Results
['Num. Dyn. Feat', 'single words', 'word pairs', 'trigrams', '4-grams']
Observe that all of the bag-of-phrases models outperform the best bag-of-words model. Figure 6: N-grams Perhaps it is a mistake, then to cut rare words and rare word-pairs, when using a bag-of-pairs model? In this experiment, no rare words or pairs were cut; only the cut for the MI was altered. Thus, somehow, when word-pairs come into play, failing to cut rare words and phrases makes things worse!