Outlier Detection for Improved Data Quality and Diversity in Dialog Systems
Table 4: Classifier accuracy when training on one dataset and testing on another (top and middle), and coverage of the test set for each training set (bottom). As expected, the highest scores are when we train and test on the same data, but off the diagonal the unique test set (gray column) is considerably harder for models trained on other data while a model trained on unique performs consistently well. This accuracy trend is matched in coverage.
['Metric', 'Training', 'Test Set same', 'Test Set random', 'Test Set unique']
As expected, the highest scores are on the diagonal—training and testing on the same source data. More importantly however, training on the unique data produces a model that is robust, performing well across all three test sets. In contrast, training on the same or random data produces models that perform substantially worse on the unique test set. This trend is also present in the coverage scores in the bottom section of the table.