Pretrained Semantic Speech Embeddings for End-to-End Spoken Language Understanding via Cross-Modal Teacher-Student Learning
Table 3: Effect of layers fine-tuning
['ASR', 'NLU', 'Accuracy on Test, %', 'Accuracy on Test, %', 'Accuracy on Test, %', 'Validation']
While it is not completely clear how many layers should be fine-tuned, we can conclusively tell that fine-tuning of former ASR encoder layers is more beneficial than former NLU layers. We decide to fine-tune the two top former ASR encoder layers. The results also illustrate that the optimization of SLU model for smaller distance of its output from the output of NLU model is general enough and translates to accuracy improvements in the downstream tasks, although not in all cases.