Attention over Parameters for Dialogue Systems
Table 2: Results for the goal-oriented responses in both MWOZ and SMD. Last raw, and italicized, are the Oracle results, and bold-faced are best in each setting (w and w/o Universal). Results are averaged among three run (full table in Appendix A6).
['[BOLD] Model', '[BOLD] F1', '[BOLD] BLEU', '[BOLD] SQL [ITALIC] Acc', '[BOLD] SQL [ITALIC] BLEU', '[BOLD] BOOK [ITALIC] Acc', '[BOLD] BOOK [ITALIC] BLEU']
1) AoP and AoR perform consistently better then other baselines which shows the effectiveness of combining parameters by using the correct prior V; 2) AoP performs consistently, but marginally, better than AoR, with the advantage of an algorithmic faster inference; 3) Using Oracle (AoP+O) gives the highest performance in all the measures, which shows the performance upper-bound for AoP. Hence, the performance gap when not using oracle attention is most likely due to the error in attention α (i.e., 2% error rate). the model performance decreases, which confirms that good inductive bias is important for learning how to select and combine different parameters (experts). By adding the layer recursion, both models are able to consistently improve all the evaluated measures, in both Persona-Chat and the Task-Oriented tasks. Especially AoP, which achieves better performance than Oracle (i.e. single layer) in the SQL accuracy, and a consistently better performance in the Persona-Chat evaluation.