Iterative Delexicalization for Improved Spoken Language Understanding
Table 1: Slot labeling F1 score comparison for different slot categories in Facebook dataset using baseline Attention BiRNN parser [Liu and Lane, 2016].
['[BOLD] Slot category', '[BOLD] Total fraction', '[BOLD] Baseline F1 %', '[BOLD] Our model F1 %']
Recently, joint intent classification and slot labeling SLU models based on recurrent neural networks has been shown to achieve state-of-the-art performance on benchmark datasets However, these models often suffer from poor slot labeling accuracy when an utterance contain slots with large semantic variability, dissimilar to those encountered during training e.g. message text, partial show/artist names. In this paper, we refer such slots as out-of-distribution (OOD) slots. The main characteristics of such OOD slots are that they do not have fixed vocabulary, and moreover can take values with length and word distribution very dissimilar to their training vocabulary. Usually they also contain many out-of-vocab words. Standard SLU models are trained on expensive labeled training datasets where OOD slots with large semantic variability are never well represented. Presence of out-of-vocabulary words and phrases have been shown to further degrade performance of RNN based SLU models We also perform some slot level error analysis to observe the advantage of our iterative delexicalization algorithm for out-of-distribution slots. For Facebook dataset, we divide the slots into two categories; first those of the form of message text, and the other non-message slots, since message texts can often have large semantic variability. We observe that our iterative delexicalization algorithm significantly improves the F1 score of message text slots while maintaining similar F1 score for other slots.