Outlier Detection for Improved Data Quality and Diversity in Dialog Systems
Table 5: Slot-filling: Diversity scores for data collected in each round (top), and the number of samples collected (bottom). The data for the All column combines the previous two sets in the row and the data from (same Round 1). As seen for intent classification, the unique approach produces data that is of considerably higher diversity.
['[EMPTY]', 'Data Collection Round 1', 'Data Collection Round 2', 'Data Collection Round 3', 'All']
As in the classifier experiment, we observe that data produced by the unique pipeline is of higher diversity than the other two pipelines.