Constructing a Natural Language Inference Dataset using Generative Neural Networks
Table 4: The performance of classifiers trained on the original and generated datasets. The classifiers were tested on original test set. The generated datasets were generated by the models from Table 3. The generated datasets were filtered with threshold 0.6.
['Dataset', 'loss', 'accuracy%']
The best generated dataset was generated by AttEmbedDecoder. The accuracy of its classifier is only 2.7 % lower than the accuracy of classifier generated on the original human crafted dataset. The comparison of the best generated dataset to the original dataset shows that the datasets had only 0.06% of identical examples. The average length of the hypothesis was 7.97 and 8.19 in the original dataset and in the generated dataset, respectively. In another experiment the generated dataset and the original dataset were merged to train a new classifier. Thus, the merged dataset contained twice as many examples as other datasets. The accuracy of this classifier was 82.0%, which is 0.8 % better than the classifier trained solely on the original training set. However, the lowest average loss is achieved by the classifier trained on the original dataset.