“None of the Above”:Measure Uncertainty in Dialog Response Retrieval
Table 3: @10 and @2 represent metrics on 10 and 2 candidates respectively. R represents recall, N represents binary NOTA classification accuracy, NF1 represents the F1 score on the NOTA class, and GF1 represents the F1 score on the ground-truth-present class. Average F1 is obtained on the 4 F1 scores.
['[EMPTY]', '50% NOTA Test Results (%) R@10', '50% NOTA Test Results (%) R@2', '50% NOTA Test Results (%) N@10', '50% NOTA Test Results (%) N@2', '50% NOTA Test Results (%) N F1@10', '50% NOTA Test Results (%) N F1@2', '50% NOTA Test Results (%) G F1@10', '50% NOTA Test Results (%) G F1@2', '50% NOTA Test Results (%) Average F1']
The direct predict model is run as it does not need further tuning. Threshold approach, especially with softmax probability as threshold, will need separate rounds of tuning on the threshold. Here, the average F1 is averaged on all 4 F1 scores. For each model architecture, the best performing setting for each metric is in bold.