Acoustic Model Optimization Based On Evolutionary Stochastic Gradient Descent with Anchors for Automatic Speech Recognition
Table 2: CE loss and WER of ESGD with anchors on BN50
['model', 'loss', 'WER']
In the first experiments, we use the SGD baseline and ESGD baseline as the initial anchors. The drop of loss is monotonic. On the left panel, it takes 5 generations to break the SGD baseline while on the right it takes 8 generations. After 20 generations, ESGD using the SGD baseline as the initial anchor has the loss of 1.935 and WER 16.5, which improves significantly over the baseline model. Note that the SGD baseline is already a decent model trained under well-tuned training recipe. On the other hand, ESGD using the ESGD baseline as the initial anchor has the loss of 1.899 and WER 16.3, also improves from the ESGD baseline. The ESGD baseline is a very strong baseline. Using anchors is able to further improve it. The ESGD optimization with anchors can be conducted in an iterative fashion. After each round of evolution (in this case 20 generations), the best model can be used as the initial anchor model for the next round. The design of the ESGD will guarantee that the new anchor model will be no worse than the previous anchor in terms of loss. Therefore, the best model can be improved iteratively. After each 20 generations of evolution, the best model is used to update the anchor and restart the evolution process.