Do Not Have Enough Data? Deep Learning to the Rescue!
Table 6: Accuracy of LAMBADA with or without label vs. unlabeled data for ATIS dataset with 5 samples per class. Significant improvement for BERT and SVM classifiers (*McNemar, p−value<0.01).
['Classifier', 'Base.', 'Unlab. Data', 'Unlab. GPT', 'LAMBADA']
Our augmentation framework does not require additional unlabeled data. As such, it can be applied when unlabeled data is unavailable or costly. To test the expected LAMBADA performance in such a scenario, we compared it to a semi-supervised approach [ruder2018strong] that uses unlabeled data. To create an unlabeled dataset, we randomly selected samples from the original dataset while ignoring their labels. Next, following a simple weak labeling approach, we classified the samples with one of the classifiers after training it on the labeled dataset. We compared LAMBADA’s classification results with the results we obtained from this classifier. Surprisingly, for most classifiers, LAMBADA achieves better accuracy compared to a simple weak labeling approach. Clearly, the generated dataset contributes more to improving the accuracy of the classifier than the unlabeled samples taken from the original dataset. We further assessed the importance of the ”generated” labels by removing them from LAMBADA’s synthesized dataset. In future work, we plan to use various data balancing approaches on the unlabeled dataset to assess the importance of the second factor above.