Hamming Sentence Embeddings for Information Retrieval
Table 1. The average absolute correlation of sentence embedding’s dimensions (Equation (1)) on the Wikipedia-1mio dataset. The compressed embeddings’ correlation is about 3-4 lower than the original embeddings’.
['embedding type', '[BOLD] avg. correlation (%) original', '[BOLD] avg. correlation (%) compressed']
To test this hypothesis, we inspect the scale of correlations between dimensions in the embeddings: Let e1,... ,en denote d-dimensional embeddings, and let ρ(i,j) (for i,j=1,... ,d) denote the correlation between two dimensions in these embeddings. We measure the overall correlation within the embeddings as (1) 1d2d∑i=1d∑j=1|ρ(i,j)|. We observe a strong reduction of correlation between the embeddings by a factor of about 3-4, which illustrates the decorrelating effect of our compressor.