Pretrained Semantic Speech Embeddings for End-to-End Spoken Language Understanding via Cross-Modal Teacher-Student Learning
Table 4: Effect of learning rate schedule
['Warmup', 'LR', 'Epochs', 'Accuracy on Test, %', 'Accuracy on Test, %', 'Accuracy on Test, %', 'Validation']
After deciding which layers to fine-tune, we run a series of experiments to determine the best learning rate schedule. When we increase number of warmup steps, we notice positive effect from slower learning rate ramp up. However, as number number of warmup steps becomes close to the total number of fine-tuning steps, we have to increase number of epochs from 10 to 20 in order to see the whole fine-tuning process.