Poly-encoders: architectures and pre-training strategies for fast and accurate multi-sentence scoring
Table 7: Bi-encoder results on the ConvAI2 valid set for different choices of function red(⋅).
['Setup', 'ConvAI2 valid Recall@1/20']
We considered three ways of reducing the output into one representation via red(⋅): choose the first output of the transformer (corresponding to the special token [S]), compute the average over all outputs or the average over the first m≤N outputs. We use the first output of the transformer in our experiments as it gives slightly better results. the results obtained for different types of reductions on top of the Bi-encoder. Specifically we compare the Recall@1/20 on the ConvAI2 validation set when taking the first output of BERT, the average of the first 16 outputs, the average of the first 64 outputs and all of them except the first one ([S]).