Improving Multi-turn Dialogue Modelling with Utterance ReWriter
Table 4: BLEU, ROUGE (F1), and EM scores on the test set. EM score is split into the results on the positive (left) and negative (right) test samples. The first half is LSTM-based models and the second half is Transformer-based. Bold denotes best results.
['[EMPTY]', 'BLEU-1', 'BLEU-2', 'BLEU-4', 'ROUGE-1', 'ROUGE-2', 'ROUGE-L', 'EM']
S5SS3SSS0Px1 Accuracy of Generation We first evaluate the accuracy of generation leveraging three metrics: BLEU, ROUGE, and the exact match score(EM) (the percentage of decoded sequences that exactly match the human references). For the EM score, we report separately on the positive and negative samples to see the difference. We report BLEU-1, 2, 4 scores and the F1 scores of ROUGE-1, 2, This implies the self-attention mechanism is helpful in identifying coreferred and omitted information. More analysis on how it helps coreference resolution can be seen in the next section. The generation mode does not work well in our setting since all words can be retrieved from either H or Un. Pointer-based models outperform the more complex generation-based and hybrid ones. Separately processing H and Un then combine their attention with a learned λ performs better than treating the whole dialogue tokens as s single input, though the improvement is less significant compared with previous two mentions. Overall our proposed model achieves remarkably good performance, with 55.84% of its generations exactly matches the human reference on the positive samples. For negative samples, our model properly copied the the original utterances in 98.14% of the cases. It suggests our model is already able to identify the utterances that do not need rewriting. Future work should work on improving the rewriting ability on positive samples.