Sequential Attention-based Network for Noetic End-to-End Response Selection
Table 5: Ablation analysis on the development set for the DSTC7 Advising dataset.
['[BOLD] Sub', '[BOLD] Models', '[BOLD] R@1', '[BOLD] R@10', '[BOLD] R@50', '[BOLD] MRR']
For Ubuntu subtask 1, ESIM achieved 0.854 R@10 and 0.6401 MRR. If we removed context’s local matching and matching composition to accelerate the training process (“-CtxDec”), R@10 and MRR dropped to 0.845 and 0.6210. Further discarding the last words instead of the preceding words for the context (“-CtxDec & -Rev”) degraded R@10 and MRR to 0.840 and 0.6174. Ensembling the above three models (“Ensemble”) achieved 0.887 R@10 and 0.6790 MRR. Ensembling was performed by averaging output from models trained with different parameter initializations and different structures. We used ESIM without context’s local matching and matching composition for computational efficiency. We observed similar trends as on the Ubuntu data set. “-CtxDec & -Rev” degraded R@10 and MRR over “-CtxDec”, yet the ensemble of the two models always produced significant gains over individual models.