Selecting Machine-Translated Data for Quick Bootstrapping of a Natural Language Understanding System
Table 3: Results of the filtering approaches. Relative changes with respect to the baseline are given in parentheses.
['[BOLD] Approach(es)', '[BOLD] Dataset size', '[BOLD] SemER (%)']
In the following, we evaluate whether our filtering and post-processing approaches can improve the quality of the MT training data further. While filtering based on semantic information yields an improvement in SemER over using MT data as they are, filtering based on MT scores only yields a slight improvement in one of the conditions. For filtering based on semantic information, our results confirm in a large-scale industry setting that training on utterances for which the intent is retained after back-translation is useful. In addition, our results show that performance can be improved further by either additionally removing utterances for which the slots are not retained or by removing utterances for which the confidence is very low. Here, removing low confidence utterances yields slightly better results. While we tested with 0.1 as a threshold, results might be improved further by optimising this threshold, potentially even per domain.