Modeling Intent, Dialog Policies and Response Adaptation for Goal-Oriented Interactions
Table 3: NLU Ablation Experiments on Meet & Greet and Simon Says Dataset
['[EMPTY]', 'Prec', 'Rec', 'F1']
To evaluate the feature explorations for the NLU model, we present an ablation study on the LSTM, BiLSTM, Transformer, USE, text + speech We use the embedding intent classifier as the baseline for the NLU experiments. For sequence models and Transformer model, we use word level features instead of the sentence-level features. The performance of the baseline model with the text + speech act features added is very similar to the baseline. Adding USE to the baseline improves the scores and this can be explained from the fact that USE, which is a transformer based model, generates better representations compared to individual text/speech act features. Consistent with the fact that BERT has shown a lot of success in several NLP tasks recently, we notice that adding BERT representations alone to the baseline improves the performance significantly. On the word-level features, we observe that while LSTM, BiLSTM and Transformer representations show slight improvement, adding BERT along with these shows a significant performance increase. The results show that adding BERT with the Transformer model helps achieving best performance on the dataset.