Utterance Pair Scoring for Noisy Dialogue Data Filtering
Table 4: Automatic evaluation results for generated responses (English). BLEU-1 and ROUGE (×100) are computed without symbols. The bold denotes the best result in the same data size in each metric.
['Training data', '# of pairs', 'len', 'distinct1', 'distinct2', '0BLEU-1 [ITALIC] s', '0bp [ITALIC] s', '0B1bp [ITALIC] s', 'BLEU-1', '0bp', '0B1bp', '0METEOR', 'ROUGE', '0CID', '0EA', '0VE', '0GM']
We first compare the datasets using the following automatic metrics: the average response length in tokens (len), the type-token ratio for {1,2}-grams (distinct-{1,2}), and two reference-based token overlap metrics which are most commonly used, BLEU-1 and ROUGE, following previous work Sordoni et al. ; Li et al. Xing et al. We especially focus on the scores on the response length and distinctiveness metrics, which point to the diversity of the generated responses. Applying our filtering method resulted in data with a higher score on distinct-{1,2} while maintaining a moderate length. This points towards our filtered data being more diverse than both non-filtered data and data filtered by other methods.