Building a Production Model for Retrieval-Based Chatbots
Table 5: Recall@k for random, frequency, and clustering whitelists of different sizes. The “+” indicates that the true response is added to the whitelist.
['[BOLD] Whitelist', '[BOLD] R@1', '[BOLD] R@3', '[BOLD] R@5', '[BOLD] R@10', '[BOLD] BLEU']
The results in Table 5 show that the three types of whitelists perform comparably to each other when the true response is added. However, in the more realistic second case, when recall is only computed on examples with a response already in the whitelist, performance on the frequency and clustering whitelists drops significantly. [CONTINUE] The BLEU scores computed with the frequency and clustering whitelists are slightly higher than those computed with random whitelists. [CONTINUE] The recall results in Table 5 seem to indicate that the clustering-based whitelists are strictly superior to the frequency-based whitelists in the realistic case when we only consider responses that are already contained in the whitelist, but this analysis fails to account for the frequency with which this is the case.