Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
Table 6: Precision, recall and F1-score on Braun et al. corpora. *Benchmark run in August 2017 by the authors of bench17 . **Benchmark run in January 2018 by the authors of this paper.
['corpus', 'NLU provider', 'precision', 'recall', 'F1-score']
For the raw results and methodology, see https://github.com/snipsco/nlu-benchmark. The main metric used in this benchmark is the average F1-score of intent classification and slot filling. The data consists in three corpora. Two of the corpora were extracted from StackExchange, one from a Telegram chatbot. The exact same splits as in the original paper were used for the Ubuntu and Web Applications corpora. At the date we ran the evaluation, the train and test splits were not explicit for the Chatbot dataset (although they were added later on). In that case, we ran a 5-fold cross-validation. For Rasa, we considered all three possible backends (Spacy, SKLearn + MITIE, MITIE), see the abovementioned GitHub repository for more details. However, only Spacy was run on all 3 datasets, for train time reasons. For fairness, the latest version of Rasa NLU is also displayed. Results show that Snips NLU ranks second highest overall.