Teaching Machines to Converse
Table 3.6: MMI-bidi gains over comparator systems, based on pairwise human judgments.
['[BOLD] Comparator', '[BOLD] Gain', '[BOLD] 95% CI']
We employed crowdsourced judges to provide evaluations for a random sample of 1000 items in the Twitter test dataset. Each output pair was ranked by 5 judges, who were asked to decide which of the two outputs was better. They were instructed to prefer outputs that were more specific (relevant) to the message and preceding context, as opposed to those that were more generic. Ties were permitted. Identical strings were algorithmically assigned the same score. The mean of differences between outputs is shown as the gain for MMI-bidi over the competing system. At a significance level of α=0.05, we find that MMI-bidi outperforms both baseline and greedy Seq2Seq systems, as well as the weaker SMT and SMT+RNN baselines. MMI-bidi outperforms SMT in human evaluations despite the greater lexical diversity of MT output.