G-DAug: Generative Data Augmentation for Commonsense Reasoning
Table 8: Test performance of an unaugmented baseline model and ensembled with finetuned GPT-2 generator on WinoGrande. We use weighted average ensemble with weights tuned on validation data.
['Method', 'Test AUC']
Generator/Task Model Ensemble. G-DAug harnesses pretrained knowledge from GPT-2 in order to improve a RoBERTa-based task model; a more standard approach to this end (albeit, with twice the computational cost at runtime) would be to ensemble the two models instead. We adopt a weighted-average ensemble method, where the weights are tuned on validation data (the tuning is important to achieve peak performance). The ensemble model performs same as the baseline model, and G-DAug-Combo outperforms both of them by 3.9 point. This suggests that G-DAug is more effective than simply ensembling the finetuned generator.