Distilling Knowledge for Fast Retrieval-based Chat-bots
Table 3. Average milliseconds to process a single test sample.
['№\xa0of candidates', '10', '100']
We demonstrate the trade-off in speed and performance between the BERT bi-encoder and our enhanced BERT cross-encoder. Time taken by the cross-encoder to process a set of candidate responses grows exponentially large as the set increases in size. In the case of BERT bi-encoders, since candidate vectors can be computed offline, increasing candidates has a negligible impact on inference time.