Differentiable Product Quantization for End-to-End Embedding Compression
Table 7: Effect of using DPQ on BERT. DPQ gives a compression ratio of 37× on the embedding table while the model’s performance on downstream tasks remains competitive.
['[BOLD] Embeddings', '[BOLD] CR', '[BOLD] Squad 1.1', '[BOLD] Squad 2.0', '[BOLD] CoLA', '[BOLD] MNLI', '[BOLD] MRPC', '[BOLD] XNLI']
To further test DPQ, we replace the embedding layer in BERT with our DPQ for both pre-training and fine-tuning. We do not perform hyper-parameter search for DPQ, but simply use the best configuration from our experiments on WMT19 EnDe using Transformer, i.e. we use DPQ-SX with no subspace-sharing with K=32,D=128.