Learning Semantic Textual Similarity from Conversations
Table 1: Precision at N (P@N) results of Reddit model with different encoders on the test set with 1 true response and 99 randomly selected negatives.
['[EMPTY]', 'P@1', 'P@3', 'P@10']
The DAN encoder (with n-grams) as used in \newciteefficientsmartreply can be considered a strong baseline. The transformer encoder outperforms DAN for all values of N. The transformer encoder achieves a P@1 metric of 65.7% while the DAN encoder achieves only 56.1%. Given its greater performance, we use a transformer encoder for the remainder of the experiments reported in this work.