Poly-encoders: architectures and pre-training strategies for fast and accurate multi-sentence scoring
Table 5: Average time in milliseconds to predict the next dialogue utterance from C possible candidates on ConvAI2. * are inferred.
['[EMPTY]', 'Scoring time (ms) CPU', 'Scoring time (ms) CPU', 'Scoring time (ms) GPU', 'Scoring time (ms) GPU']
The difference in timing between the Bi-encoder and the Poly-encoder architectures is rather minimal when there are only 1000 candidates for the model to consider. The difference is more pronounced when considering 100k candidates, a more realistic setup, as we see a 5-6x slowdown for the Poly-encoder variants. Nevertheless, both models are still tractable. The Cross-encoder, however, is 2 orders of magnitude slower than the Bi-encoder and Poly-encoder, rendering it intractable for real-time inference, e.g. when interacting with a dialogue agent, or retrieving from a large set of documents. Thus, Poly-encoders, given their desirable performance and speed trade-off, are the preferred method.