How to Evaluate the Next System: Automatic Dialogue Evaluation from the Perspective of Continual Learning
Table 3: The plasticity for evaluating dialogue systems using fine-tuning, EWC, VCL in time order. The lowercase letters are short for dialogue systems, and r, a, k, c, h indicate retrieval-system, attention-system, keywords-system, CVAE-system, human-system respectively.
['[EMPTY]', 'fine-tuning  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'fine-tuning  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'fine-tuning  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'fine-tuning  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'fine-tuning  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'EWC  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'EWC  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'EWC  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'EWC  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'EWC  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'VCL  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'VCL  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'VCL  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'VCL  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h', 'VCL  [ITALIC] r→ [ITALIC] a→ [ITALIC] k→ [ITALIC] c→ [ITALIC] h']
Different from fine-tuning, continual learning (EWC and VCL) add restrictions on the learning of the next task, which would reduce the interference on the previous task but affect the learning capacity of the new task. For example, after learning from human-system, the accuracy on human-system of fine-tuning based dialogue evaluator is the highest 0.380, while the accuracy on CVAE-system decreases sharply from 0.345 to 0.190. Although for EWC and VCL based dialogue evaluators, the accuracy on CVAE-system is only slightly affected, but the plasticity of human-system decrease. This is the stability-plasticity dilemma. The EWC based evaluator improves the stability meanwhile maintaining the plasticity. To some extent, we consider that EWC based dialogue evaluator achieves a trade-off between stability and plasticity and alleviates the stability-plasticity dilemma.