Curriculum-based transfer learning for an effective end-to-end spoken language understanding and domain portability
Table 4: Comparison between state-of-the art pipeline approach and end-to-end system on the MEDIA test
['System', 'CER', 'CVER']
In order to evaluate the contribution of semantic, syntactic and morphological features, we choose to design two templates: one considering only the surface form of the word, and the other one considering all available features. Systems based on these two templates are respectively denoted SFCRFlex and SFCRFlex+feat. These systems process slot filling on automatic transcriptions. For our experiments, we feed them with outputs from our end-to-end ASR system (with LM rescoring) fine tuned one the MEDIA training data. The pipeline approach reaches a CER of 16.1% and a CVER of 20.9% that is slightly better than the results reached by the end-to-end approach. By computing the 95% confidence interval through the Student’s t-test, we can observe than the confidence margin is 0.7 for CER (0.8 for CVER). By the way, this means that the differences between the pipeline and the end-to-end approaches are not statistically significant. Another remark comes from the results got with SFCRFlex: this system uses only lexical information and can be directly compared to the end-to-end approach that actually does not use external features coming from human expertise or NLP tools like predefined semantic categorie, POS, word governor and dependencies… In this case, the comparison is largely at the advantage of the end-to-end system. This shows that it would be important to investigate in the future how to inject to the end-to-end model external information that help so much in the pipeline approach.