Controlling Risk of Web Question Answering
Table 6. Comparison results over the probe number.
['SogouRC', 'PROBE-CNN [ITALIC] P( [ITALIC] T)', 'AURC 14.60', 'ROC 75.12', 'AP 53.1']
We further conduct experiments to analyze our probe-based qualify model. We first investigate the effect of layer numbers to our PROBE-CNN method. In our original design, we probe each intermediate layer of the MRC model to obtain signals for predictive uncertainty estimation. Here we test the performance if we only probe the last layer P(T) of the MRC model, namely PROBE-CNNP(T). We can see that there is a clear performance drop if we only probe the last layer. The relative decrease of PROBE-CNNPT over the vanilla PROBE-CNN is about 3% in terms of AURC on the SogouRC dataset. The results demonstrate that the last layer does contain some useful signals, e.g., the distribution of the final s and e scores, but the intermediate layers can bring much richer information for uncertainty estimation.