Building a Production Model for Retrieval-Based Chatbots
Table 7: Results of the human evaluation of the responses produced by our model. A response is acceptable if it is either good or great. Note: Numbers may not add up to 100% due to rounding.
['[BOLD] Whitelist', '[BOLD] Great', '[BOLD] Good', '[BOLD] Bad', '[BOLD] Accept']
The results of the human evaluation are in Table 7. Our proposed system works well, selecting acceptable (i.e. good or great) responses about 80% of the time and selecting great responses more than 50% of the time. Interestingly, the size and type of whitelist seem to have little effect on performance, indicating that all the whitelists contain responses appropriate to a variety of conversational contexts.