Outlier Detection for Improved Data Quality and Diversity in Dialog Systems
Table 1: Outlier detection effectiveness for error detection in an artificial setting (constructed by randomly adding content from other intents) and a real setting (manually checked utterances from a crowdsourced set). The artificial results are represented by different values of p (1%, 2%, 4%, and 8%), where p represents different amounts of errors injected into each intent class. Our proposed neural methods are consistently more effective, reducing the manual effort required to identify errors.
['Method', '[EMPTY]', '1%', 'MAP 2%', 'MAP 4%', 'MAP 8%', 'Real', 'Recall@ [ITALIC] k=10% 1%', 'Recall@ [ITALIC] k=10% 2%', 'Recall@ [ITALIC] k=10% 4%', 'Recall@ [ITALIC] k=10% 8%', 'Recall@ [ITALIC] k=10% Real']
The neural methods outperform the baselines in both settings, demonstrating the effectiveness of our proposed approach. However, the relative performance of the neural methods differs substantially between the two settings. Specifically, (1) SIF performs better than an unweighted average on artificial data, but on real data we see the opposite trend, (2) combining rankings with Borda appears to help on the artificial data, but not on the real data, (3) ranking by length is surprisingly effective on the real data, and (4) results tend to be lower on the real data than the artificial (even at lower values of p). This last point suggests that the commonly used artificial setting does not perfectly capture the types of errors that occur in practice. The effectiveness of the neural methods, and USE in particular, is again clear. In the real data, when considering just the first 20% of the list, USE covers over 85% of the errors on average. One easy example was “No checkbox, more?” when the intent was to order more checks. This is clearly an error, which would at the very least need to have checkbox replaced by checkbook. In contrast, one hard example for USE was “How much money do my banks” when the intent was to request the user’s balance. Until the last word, this example looks like it will be a valid balance request. These examples show that the system is qualitatively fitting our expectations for error detection.