Zero-shot User Intent Detection via Capsule Neural Networks
Table 4: Zero-shot intention detection results using IntentCapsNet-ZSL on two datasets. All the metrics (Accuray, Precision, Recall and F1) are reported using the average value weighted by their support on per class.
['[BOLD] Model', '[BOLD] SNIPS-NLU (on 2 emerging intents) Accuracy', '[BOLD] SNIPS-NLU (on 2 emerging intents) Precision', '[BOLD] SNIPS-NLU (on 2 emerging intents) Recall', '[BOLD] SNIPS-NLU (on 2 emerging intents) F1', '[BOLD] CVA (on 20 emerging intents) Accuracy', '[BOLD] CVA (on 20 emerging intents) Precision', '[BOLD] CVA (on 20 emerging intents) Recall', '[BOLD] CVA (on 20 emerging intents) F1']
CMT has higher precision but low accuracy and recall on the SNIPS-NLU dataset. CDSSM fails on CVA dataset, probabily because the character-level model is suitable for English corpus but not for CVA, which is in Chinese. Ablation Study. “w/o Self-attention” is the model without self-attention: the last forward/backward hidden states of the bi-LSTM recurrent encoder are used; “w/o Bi-LSTM” uses the LSTM with only a forward pass; “w/o Regularizer” does not encourage discrepancies among different self-attention heads: it adopts α=0 in the loss function. On the SNIPS-NLU dataset, each of the three modules has a comparable contribution to the whole model (around 2-3% improvement in F1 score). While on the CVA dataset, the self-attention plays the most important role, which gives the model a 5.2% improvement in F1 score.