Conducting sparse feature selection on arbitrarily long phrases in text corpora with a focus on interpretability
Table 4: Word list characteristics of different rescaling norms. Length is length of list, mean size is average number of words in the selected phrases, frequency refers to phrase occurrence in the corpus. Mean C is the average regularization value.
['method', 'time (avg sec)', 'list length (avg)', 'phrase length (avg)', 'word freq (avg)', 'word freq (median)', 'C (avg)']
As anticipated, different values of q produce lists of different quality. Higher q corresponds to lists with relatively more common words and phrases. Low q produces lists that tend to have phrases with more words. For the q=1.2 list, more than half the phrases were 3 words or longer. We also found that the length of the list increased with q. For q=4 we had longer lists with more common words. For low q the algorithm selected very targeted phrases, and not too many of them. The regularization parameter C is not comparable across q. Therefore we recalculated it via the permutation approach for each run and value of q. Mean values for C are shown on the table for reference.