“None of the Above”:Measure Uncertainty in Dialog Response Retrieval
Table 1: Results on 10 candidates. R represents recall, N represents binary NOTA classification accuracy, NF1 represents the F1 score on the NOTA class, and GF1 represents the F1 score on the ground-truth-present class. Average F1 is the average of NF1 and GF1.
['[EMPTY]', '[ITALIC] R10', '[ITALIC] N10', '[ITALIC] NF110', '[ITALIC] GF110', 'Average F1']
Due to space limitation, this table only displays results on 10 candidates. Complete results on other numbers of candidates, which have similar performance patterns as 10, are found in the Appendix. The thresholds and hyperparameters are tuned on the validation set according to the highest average F1 score. For the selection model, in addition to the original dataset, we also train the model on a modified training dataset, containing _NOTA choices as in inference datasets, with the same set of hyperparameters. As expected, since there are now fewer real distractor responses , training including _NOTA improves the model’s NOTA classification performance, but sacrifices recall scores, which is not desirable. In all the models, regardless of the training dataset used and the model architecture, adding a logistic regression on top of the LSTM output significantly improves average F1 scores. Specifically, the highest F1 scores are always achieved with logits scores as LogReg input features. These results show that, though setting a threshold is a common heuristic to balance true and false acceptance rates Larson et al. This finding is underlined by receiver operating characteristic (ROC) curves on the validation set