A Manually Annotated Chinese Corpus forNon-task-oriented Dialogue Systems
Table 4: Comparison results (%). Higher scores indicate better results. Cut@N: responses with rating ≥N are considered as positive, and as negative otherwise. Larger cut indicates a stricter standard. Best results in each column is marked as bold.
['[BOLD] Category', '[BOLD] Models', '[BOLD] Cut@ [BOLD] 3 P@1', '[BOLD] Cut@ [BOLD] 3 MAP', '[BOLD] Cut@ [BOLD] 3 MRR', '[BOLD] Cut@ [BOLD] 4 P@1', '[BOLD] Cut@ [BOLD] 4 MAP', '[BOLD] Cut@ [BOLD] 4 MRR', '[BOLD] Cut@ [BOLD] 5 P@1', '[BOLD] Cut@ [BOLD] 5 MAP', '[BOLD] Cut@ [BOLD] 5 MRR']
We follow the paradigm of question answering to separate responses to be “positive” and “negative” when evaluating the ranked responses given a prompt. In doing so, we set rating thresholds at 3 ,4 and 5, where responses with gold-standard rating ≥N are considered as positive instances, and otherwise as negative instances. Therefore, larger N indicates stricter standard. In particular, we remove all responses of a prompt in evaluation if none of them is considered positive under a specific cut, because for this prompt, all models would score 0 no matter how its responses are ranked.