Durkheim Project Data Analysis Report
Table 14: All-Word-Pair Cuts
['[EMPTY]', 'Mean Accuracy', 'Std. Dev.']
By selecting corpus pairs, thousands of the highest-MI pairs are discarded, as well as most of the low-MI pairs, as well. Perhaps it is possible to replicate the corpus-pairs results by applying a simple cut to the all-pairs dataset, and merely discarding the low-MI pairs? This table compares a bag-of-words model to several different all-pairs models, with different MI cuts applied. Including all pairs does not improve the score over the bag-of-words. By cutting out low-MI pairs, the score can be improved somewhat, but the effect is not dramatic; certainly not as strong as the decision to use corpus-pairs.