How to Evaluate the Next System: Automatic Dialogue Evaluation from the Perspective of Continual Learning
Table 1: Experimental results for evaluating dialogue systems.
['order', 'methods', 'Plasticity  [ITALIC] t=1', 'Plasticity  [ITALIC] t=2', 'Plasticity  [ITALIC] t=3', 'Plasticity  [ITALIC] t=4', 'Plasticity  [ITALIC] t=5', 'Stability  [ITALIC] t=2', 'Stability  [ITALIC] t=3', 'Stability  [ITALIC] t=4', 'Stability  [ITALIC] t=5']
On the plasticity (generalization) side, fine-tuning and EWC achieve the best performance while stationary learning is the worst, proving our motivation on the generalization issue of existing stationary dialogue evaluators. Retraining is lower than fine-tuning, which we believe is because retraining cares all systems but fine-tuning only concerns the current ones. It is interesting to find that individual learning is not as good as fine-tuning, which we think the reason is that fine-tuning can benefit from the knowledge of previous systems, so do EWC and VCL.