Controlling Risk of Web Question Answering
Table 5. Comparison results on the answerable datasets.
['[EMPTY]', '[EMPTY]', 'SogouRC Answerable AURC', 'SogouRC Answerable ROC', 'SogouRC Answerable AP', 'SQuAD 1.0 AURC', 'SQuAD 1.0 ROC', 'SQuAD 1.0 AP']
Since the UD instances are the most difficult to handle in the risk control framework, some people might wonder whether our method gain the improvement by just better recognizing those UD instances from the rest. Here we conduct a further analysis by comparing all the four risk control methods only on the answerable queries. We use the answerable part in SoguoRC (Named SogouRC Answerable) and SQuAD 1.0 dataset for experiments. From the results, we can see that our method can also consistently outperform all the baseline methods in terms of all the evaluation metrics. The results indicate that our PROBE-CNN can well control the risk via distinguishing AD+ from AD−.