Poly-encoders: architectures and pre-training strategies for fast and accurate multi-sentence scoring
Table 3: Validation performance (R@1/20) on ConvAI2 using pre-trained weights of BERT-base with different parameters fine-tuned. Average over 5 runs (Bi-encoders) or 3 runs (Cross-encoders).
['Fine-tuned parameters', 'Bi-encoder', 'Cross-encoder']
The learning rate is initialized to 5e-5 with a warmup of 100 iterations for Bi- and Poly-encoders, and 1000 iterations for the Cross-encoder. The learning rate decays by a factor of 0.4 upon plateau of the loss evaluated on the valid set every half epoch. Fine-tuning the entire network is important, with the exception of the word embeddings.