Teaching Machines to Converse
Table 6.2: AdverSuc and machine-vs-random scores achieved by different training/decoding strategies.
['Model', '[ITALIC] AdverSuc', '[ITALIC] machine-vs-random']
Higher values of AdverSuc and machine-vs-random are better. What first stands out is decoding using sampling (as discussed in Section 4.3), achieving a significantly higher AdverSuc number than all the rest models. However, this does not indicate the superiority of the sampling decoding model, since the machine-vs-random accuracy is at the same time significantly lower. This means that sampled responses based on Seq2Seq models are not only hard for an evaluator to distinguish from real human responses, but also from randomly sampled responses. A similar, though much less extreme, effect is observed for MMI−p(y), which has an AdverSuc value slightly higher than Adver-Reinforce, but a significantly lower machine-vs-random score.