Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations
Table 2: Monolingual results. Brackets show the different power means that were applied to all individual embeddings. † we normalized the embeddings of our full model with the z-norm as proposed by LeCun:1998.
['[BOLD] Model', 'Σ', '[BOLD] AM', '[BOLD] AC', '[BOLD] CLS', '[BOLD] MR', '[BOLD] CR', '[BOLD] SUBJ', '[BOLD] MPQA', '[BOLD] SST', '[BOLD] TREC']
The results show that we can substantially improve sentence embeddings when concatenating multiple word embedding types. All four embedding types concatenated achieve 2pp improvement over the best individual embeddings (GV). Incorporating further power means also substantially improve performances. GV improves by 0.6pp on average, GN by 1.9pp, MS by 2.1pp and AR by 3.7pp when concatenating p=±∞ to the standard value p=1 (dimensionality increases to 900). The combination of concatenation of embedding types and power means gives an average improvement of 3pp over the individually best embedding type.