Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
Table 7: Precision, recall and F1-score averaged on all slots and on all intents of an in-house dataset, run in June 2017.
['NLU provider', 'train size', 'precision', 'recall', 'F1-score']
In this experiment, the comparison is done separately on each intent to focus on slot filling (rather than intent classification). The main metric used in this benchmark is the average F1-score of slot filling on all slots. Three training sets of 70 and 2000 queries have been drawn from the total pool of queries to gain in statistical relevance. Validation sets consist in 100 queries per intent. Five different cloud-based providers are compared to Snips NLU (Microsoft’s Luis, API.AI now Google’s Dialogflow, Facebook’s Wit.ai, and Amazon Alexa). For more details about the specific methodology for each provider and access to the full dataset, see https://github.com/snipsco/nlu-benchmark. Each solution is trained and evaluated on the exact same datasets. Snips NLU is as accurate or better than competing cloud-based solutions in slot filling, regardless of the training set size.