Don’t Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors
Table 3: Mean Pearson correlation on STS tasks for DynaMax and averaged word vector, all using normalised word vectors. Bolded values indicate best results per task and word vector type. Note that DynaMax still outperforms avg-cos across the board, but both approaches lose to their unnormalised counterparts as reported in Table 1.
['[BOLD] Vectors', '[BOLD] Approach', '[BOLD] STS12', '[BOLD] STS13', '[BOLD] STS14', '[BOLD] STS15', '[BOLD] STS16']
For completeness, let us insist on the range [0,1] and choose sim(w,u(j)) to be the clipped cosine similarity max(0,cos(w,u(j))). This is in fact equivalent to simply normalising the word vectors. Indeed, the dot product and cosine similarity become the same after normalisation, and max-pooling with the zero vector removes all the negative values, so the resulting representation is guaranteed to be a [0,1]-fuzzy set. we can draw two conclusions. Namely, DynaMax still outperforms avg-cos by a large margin even when word vectors are normalised. However, normalisation hurts both approaches and should generally be avoided.