Robust Reading Comprehension with Linguistic Constraints via Posterior Regularization
TABLE II: Performance comparison with different settings and on different test subsets. Overall means the original, SDA and SEA test examples are evaluated together.
['Base Model', 'Train set', 'Method', 'Test set Original', 'Test set Original', 'Test set SDA', 'Test set SDA', 'Test set SEA', 'Test set SEA', 'Overall', 'Overall']
First, we investigated whether linguistic constraints can benefit RC models. (a-1) These drops indicate that base RC models cannot handle the over-confidence issue (for SDA examples) or over-sensitivity issue (for SEA examples). (a-2) Compared to other settings, Ori-Training performs best on the original test examples but has the worst overall performance because it overfits the original data and lacks of robustness to the adversarial data. (b-1) Adv-Training obtains better overall performances compared to Ori-Training (62.5→67.8), showing that the model becomes more robust when trained with adversarial examples. (b-2) Compared to Ori-Training , Adv-Training improves the performance on SDA examples (41.9→69.0) but degrades that on SEA examples (67.0→63.6). The inconsistency indicates Adv-Training cannot balance the over-confidence and over-sensitivity issues simultaneously. (c) Compared to Ori-/Adv-Training, our PR method achieves the best overall performance (62.5/67.8 vs. 71.9), verifying the effectiveness of linguistic constraint regularization. Our method can improve the performance on SDA and SEA examples simultaneously, as the constraints account for two types of adversarial examples at the same time. (d) Our PR method improves the robustness of both lightweight models (R-Net and MemReader) and the BERT model, manifesting the linguistic constraint regularization is a versatile method and has a positive effect for all the tested models. Second, we compared the effect of different ways to incorporate linguistic knowledge. For one reason, regularization on the output distribution is more straightforward than feeding traditional feature vectors into the input layer since the symbolic features may vanish after massive nonlinear operations in neural networks. For another reason, PR applies paired features, which are more informative than features assigned to individual words. For example, feature “1790s≠1780s” is more useful than features {“1790s /r/IsA/ DATE”, “1780s /r/IsA/ DATE”}.