Fast and Scalable Expansion of Natural Language Understanding Functionality for Intelligent Agents
Table 2: Results for around 200 custom developer domains. For F1, higher values are better, while for SER lower values are better. * denotes statistically significant SER difference compared to both baselines.
['Approach', '[ITALIC] F1 [ITALIC] Intent Mean', '[ITALIC] F1 [ITALIC] Intent Median', '[ITALIC] F1 [ITALIC] Slot Mean', '[ITALIC] F1 [ITALIC] Slot Median', '[ITALIC] SER Mean', '[ITALIC] SER Median']
For the custom domain experiments, we focus on a low resource experimental setup, where we assume that our only target training data is the data provided by the external developer. We report results for around 200 custom domains, which is a subset of all domains we support. For training the baselines, we use the available data provided by the developer for each domain, e.g., example phrases and gazetteers. This training data size was selected empirically based on baseline model accuracy. The generated utterances may contain repetitions for domains where the external developer provided a small amount of example phrases and few slot values per gazetteer. For the proposed method, we pre-train a DNN model on 4 million utterances and fine tune it per domain using the 50K grammar utterances of that domain and any available gazetteer information (for extracting gazetteer features). This suggests that the baseline DNN models (without pretraining) cannot be trained robustly without large available training data. The proposed pre-trained DNN significantly outperforms both baselines across all metrics (paired t-test, p