Embedding Compression with Isotropic Iterative Quantization
Table 2: Word Similarity Results.
['[EMPTY]', '[BOLD] Method', '[BOLD] MEN', '[BOLD] MTurk', '[BOLD] RG65', '[BOLD] RW', '[BOLD] SimLex999', '[BOLD] TR9856', '[BOLD] WS353']
The performance of IIQ degrades as the compression ratio increases. This is expected, since a higher compression ratio leads to more loss of information. In addition, our IIQ method consistently achieves better results than ITQ, DCCL, NLB and the pruning method. Particularly, one sees that on the Men data set, IIQ even outperforms the baseline embedding Glove. Another observation is that on TR9856, a higher compression ratio surprisingly yields better results for IIQ. We speculate that the cause is the multi-word term relations unique to TR9856. Interestingly, the pruning method results in negative correlation in SimLex999 for the GloVe embedding. This means that pruning too many small values inside word embedding can drastically destroy the embedding quality.