Efficient Large-Scale Domain Classification with Personalized Attention
Table 3: Top-N prediction accuracy (%) on the full skill set (Full) and only enabled skills (Enabled).
['[EMPTY]', 'Top-1', 'Top-3', 'Top-5']
S5SS2SSS0Px3 Behavior of Attention Mechanism Our expectation is that the model is able to learn to attend the relevant skills during the inference process. To study the behavior of the attention layer, we compute the top-N prediction accuracy based on the most relevant skills defined by the attention weights. When the model attends to the entire set of 1500 (Full), the top-5 prediction accuracy is 20.41%, which indicates that a large number of skills can process the utterance, and thus it is highly likely to miss the correct one in the top-5 predictions. This ambiguity issue can be significantly improved by users’ enabled domain lists as proved by the accuracies (Enabled): 85.62% for top-1, 96.15% for top-3, and 98.06% for top-5. Thus the attention mechanism can thus be viewed as an initial soft selection which is then followed by a fine-grained selection at the classification stage.