G-DAug: Generative Data Augmentation for Commonsense Reasoning
Table 4: Results on ARC-Challenge Scientific QA in the closed-book setting. “Synonym” refers to the synonym replacement set, “TF:failure” refers to the TextFooler failure rate, and “TF:perturbation” refers to TextFooler average perturbation ratio. G-DAug-Influence performs the best on the validation, test and synonym replacement sets, while G-DAug-Diversity is the most robust against TextFooler.
['Method', 'Validation', 'Test', 'Synonym', 'TF:failure', 'TF:perturbation']
Unlike the commonsense domain, valid science questions are harder to generate because their semantics is very precise. Despite that, G-DAug still improves on both validation and test sets by a large margin compared to the baselines. G-DAug-Influence achieves the best accuracy on the validation, test and synonym replacement sets, while G-DAug-Diversity is the most robust against the adversarial attacks but has worse accuracy than G-DAug-Rand. This suggests that optimizing for question quality is more important when the synthetic data is more noisy as it is for ARC, and that optimizing for diversity in this case can hurt performance.