Decoding Decoders: Finding Optimal Representation Spaces For Unsupervised Similarity Tasks
Table 4: Performance of different architectures and sentence representations on unsupervised similarity tasks using the cosine similarity between two vectors as the measure of their similarity. On each task, the highest performing setup for each encoder type is highlighted in bold and the highest performing setup overall is underlined. All reported values indicate Pearson/Spearman correlation coefficients for the task. RNN encoder: Using the raw encoder output (rnn-rnn) achieves the lowest performance across all tasks. Unrolling the rnn decoders dramatically improves the performance across all tasks compared to using the raw encoder rnn output, validating the theoretical justification presented in subsection 3.3. bow encoder: We do not observe the same uplift in performance from unrolling the rnn encoder compared to the encoder output; fully understanding this behaviour is left for future work.
['[BOLD] Encoder', '[BOLD] Decoder', '[BOLD] STS12', '[BOLD] STS13', '[BOLD] STS14', '[BOLD] STS15', '[BOLD] STS16']
Switching from using cosine similarity to dot product transitions from considering only angle between two vectors, to also considering their length. Empirical studies have indicated that the length of a word vector corresponds to how sure of its context the model that produces it is. This is related to how often the model has seen the word, and how many different contexts it appears in (for example, the word vectors for “January” and “February” have similar norms, however, the word vector for “May” is noticeably smaller)