QnAMaker: Data to Bot in 2 Minutes
Table 1. Retrieval And Ranking Measurements
['Domain', 'Number of QAs', 'Avg Questions per QA', 'AUC (%)', '[ITALIC] F1']
QnAMaker is not domain-specific and can be used for any type of data. To support this claim, we measure our system’s performance for datasets across various domains. The evaluations are done by managed judges who understands the knowledge base and then judge user queries relevance to the QA pairs (binary labels). Each query-QA pair is judged by two judges. We filter out data for which judges do not agree on the label. Chit-chat in itself can be considered as a domain. Hybrid of deep learning(CDSSM) and machine learning features give our ranking model low computation cost, high explainability and significant F1/AUC score. Based on QnAMaker usage, we observed these trends: Around 27% of the knowledge bases created use pre-built persona-based chitchat, out of which, ∼4% of the knowledge bases are created for chit-chat alone. The highest used personality is Professional which is used in 9% knowledge bases. Around ∼25% developers have enabled active learning suggestions. The acceptance to reject ratio for active learning suggestions is 0.31. 25.5% of the knowledge bases use one URL as a source while creation. ∼41% of the knowledge bases created use different sources like multiple URLs. 15.19% of the knowledge bases use both URL and editorial content as sources. Rest use just editorial content.