Differentiable Product Quantization for End-to-End Embedding Compression
Table 4: Comparison of DPQ against recently proposed embedding compression techniques on the PTB LM task (LSTMs with three model sizes are studied). Metrics are perplexity (PPL) and compression ratio (CR).
['[BOLD] Method', '[BOLD] Small  [BOLD] PPL↓', '[BOLD] Small  [BOLD] CR↑', '[BOLD] Medium  [BOLD] PPL↓', '[BOLD] Medium  [BOLD] CR↑', '[BOLD] Large  [BOLD] PPL↓', '[BOLD] Large  [BOLD] CR↑']
Comparison with more baselines on LM. Here we compare DPQ against the recently proposed embedding compression methods. We find that 1) other than DPQ, only Chen′18+ which requires an extra distillation procedure is able to achieve similar perplexity scores as the full embedding baseline; 2) DPQ variants (particularly DPQ-SX) are able to obtain extremely competitive perplexity scores in all cases, while offering compression ratios that are an order of magnitude larger than all the other alternatives (and is trained end-to-end in a single stage).