Durkheim Project Data Analysis Report
Table 11: Data Cuts
['Cut', 'thresh=1 Mean', 'thresh=1 Std. Dev.', 'thresh=3 Mean', 'thresh=3 Std. Dev.']
A different measure of the importance of infrequent words can be obtained by excluding them from model building: that is, by creating models from word lists that include only those words that occur 2 or 4 or more times in the text. Superficially, this seems like a wise idea. If a word appears in only one patient record, and it is found during the training phase, then it is impossible that this word will also appear in one of the test patient records. Thus, it cannot contribute to the accuracy of the model on the test set: both the positive and negative cohorts will be missing this word; it has no predictive value. If a word appears in only two patient records, then it is unlikely that one of those locations will be in the test-set (due to the 4/5’ths - 1/5’th split). Thus, one might also believe that such words have little or no predictive value. Perhaps accuracy can be increased by cutting down the dataset, and discarding all words that appear fewer than M times in the dataset. But this is very much not the case. Perhaps it is a mistake, then to cut rare words and rare word-pairs, when using a bag-of-pairs model? In this experiment, no rare words or pairs were cut; only the cut for the MI was altered. Thus, somehow, when word-pairs come into play, failing to cut rare words and phrases makes things worse!