Constructing a Natural Language Inference Dataset using Generative Neural Networks
Table 5: Discrimination error rate on the development set and a sample of 200 examples, evaluated by the discriminative model and human evaluator
['Gen. Model', 'Dev. set Disc. Model', 'Dev. set Human', 'Sample Disc. Model', 'Sample Human']
Human evaluation confirms that AttEmbedDecoder hypotheses are more difficult to separate from the original one than the hypotheses of VaeEncoderDecoder. The discriminative model discriminates better than the human evaluator. This may be due to the fact that the discriminative model has learned from a large training set, while the human was not shown any training examples. Human evaluation has shown that generated hypotheses are positively recognized if they contain a grammatical or semantic error. But even if the generated hypothesis does not contain these errors, it sometimes reveals itself by not being as sophisticated as the original example. On the other hand, the discriminative model does not always recognize these discrepancies. It relies more on the differences in distributions learned form a big training set. The true number of non-distinguishable examples may be even higher than indicated by the human discriminator error rate since the human may have correctly guessed some of the examples he could not distinguish.