FastText.zip:Compressing text classification models
Table 1: Best ranked words w.r.t. entropy (left) and norm (right) on the Amazon full review dataset. We give the rank for both criteria. The norm ranking filters out words carrying little information.
['word .', 'Entropy 1', 'Norm 354', 'word mediocre', 'Entropy 1399', 'Norm 1']
S4SS1SSS0Px2 Pruning. We fix k=d/2 and use different pruning thresholds. NPQ offers a compression rate of ×10 compared to the full model. As the pruning becomes more agressive, the overall compression can increase up up to ×1,000 with little drop of performance and no additional overhead at test time. In fact, using a smaller dictionary makes the model faster at test time. Even though fastText with the default setting uses more memory, NPQ is already on par with CNNs’ memory usage. Note that CNNs are not quantized, and it would be worth seeing how much they can be quantized with no drop of performance. Such a study is beyond the scope of this paper.