Skip-gram Language Modeling Using Sparse Non-negative Matrix Probability Estimation
Table 4: Metafeature ablation experiments on SNM5
['Ablated feature', 'PPL']
To find out how much, if anything at all, each metafeature contributes to the adjustment function, we ran a series of ablation experiments in which we ablated one metafeature at a time. When we experimented on SNM5, we found, unsurprisingly, that the most important metafeature is the feature-target count. At first glance, it does not seem to matter much whether the counts are stored in 1 or 2 buckets, but the second bucket really starts to pay off for models with a large number of singleton features e.g. This is not the case for the feature counts, where having a single bucket is always better, although in general the feature counts do not contribute much. In any case, feature counts are definitely the least important for the model.