Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds
Table 2: Performance with various labeling rates and methods with unlabeled dataset comprising 50k samples. “Dev” denotes the development set, and “Test” denotes the test set, with exact measure (EM) and F1 metric. Results from all approaches apart from the proposed one are taken from Yang et al. (2017).
['Labeling rate', 'Method', 'Dev F1', 'Test F1', 'Test EM']
In this experiment we study the performance in a semi-supervised setup at different labeling rates. For the sake of transparency, we conducted experiments following the protocol of Yang et al. Specifically, we evaluate the performance at labeling rate of 10%, 20%, 50% and 90% of the 50k unlabeled data corpus with remaining 10% used for testing. As it can be seen in the table, the proposed approach outperforms the state-of-the-art semi-supervised QA approach of Yang et al. The larger the labeling rate is, the larger the margin. However, it should be noted a part of the margin can be attributed to the approach of Yang et al. The results further suggest that the approach at a labeling rate of around 50% already starts to saturate. That is, the gain in accuracy beyond that labeling rate drops significantly, indicating the high quality and diversity of the generated questions. Generally, using the proposed approach in a semi-supervised setting seems feasible given that even at very low labeling rates such as 10%, the proposed approach reaches good accuracy not too far off from the upper bound.