Differentiable Product Quantization for End-to-End Embedding Compression
Table 8: Comparison of end-to-end DPQ against using PQ to reconstruct the embedding table after the model is trained, for the NMT task on WMT19 (En-DE).
['[BOLD] Method', '[BOLD] BLEU↑', '[BOLD] Compr. Ratio↑']
We can see that PQ degenerates the performance significantly. This is expected as small approximation errors in the embedding layer accumulate and can be amplified as the errors propagate through the deep neural nets, finally leading to large errors in the output space. End-to-end training does not have this problem as the whole system is jointly learned, so the networks can account for small approximation errors in the early layers.