Joint Online Spoken Language Understanding and Language Modeling with Recurrent Neural Networks
Table 1: ATIS Test set results on intent detection error, slot filling F1 score, and language modeling perplexity. Related joint models: RecNN: Joint intent detection and slot filling model using recursive neural network [Guo et al.2014]. RecNN+Viterbi: Joint intent detection and slot filling model using recursive neural network with Viterbi sequence optimization for slot filling [Guo et al.2014].
['[EMPTY]', '[BOLD] Model', '[BOLD] Intent Error', '[BOLD] F1 Score', '[BOLD] LM PPL']
Row 3 to row 5 are the independent training model results on intent detection, slot filling, and language modeling. The basic joint model uses a shared representation for all the three tasks. It gives slightly better performance on intent detection and next word prediction, with some degradation on slot filling F1 score. If the RNN output ht is connected to each task output directly via linear projection without using MLP, performance drops for intent classification and slot filling. Thus, we believe the extra discriminative power introduced by the additional model parameters and non-linearity from MLP is useful for the joint model. It is evident that the recurrent intent context helps the next word prediction, reducing the language model perplexity by 9.4% from 11.27 to 10.21. The contribution of local intent context to next word prediction is limited. We believe the advantageous performance of using recurrent context is a result of modeling predicted intent history and intent variations along with the growing word sequence. For intent classification and slot filling, performance of these models with intent context is similar to that of the basic joint model. Comparing to the basic joint model, the introduced slot label context (both local and recurrent) leads to a better language modeling performance, but the contribution is not as significant as that from the recurrent intent context. Moreover, the slot label context reduces the intent classification error from 2.02 to 1.68, a 16.8% relative error reduction. From the slot filling F1 scores in row 10 and row 11, it is clear that modeling the slot label dependencies by connecting slot label output to the recurrent state is very useful. As can be seen from the results, the joint model that utilizes two types of recurrent context maintains the benefits of both, namely, the benefit of applying recurrent intent context to language modeling, and the benefit of applying recurrent slot label context to slot filling. Another observation is that once recurrent context is applied, the benefit of adding local context for next word prediction is limited. It might hint that the most useful information for the next word prediction can be well captured in the RNN state, and thus adding explicit dependencies on local intent class and slot label is not very helpful.