Acoustic Model Optimization Based On Evolutionary Stochastic Gradient Descent with Anchors for Automatic Speech Recognition
Table 1: Hyper-parameters of ESGD with anchors for BN50 and SWB300
['[BOLD] params', '[BOLD] BN50  [BOLD] SGD', '[BOLD] BN50  [BOLD] ADAM', '[BOLD] SWB300  [BOLD] SGD', '[BOLD] SWB300  [BOLD] ADAM']
The size of the parent population is 100 and the offspring population 400. In ESGD, the hyper-parameters of optimizers which are to be chosen to create the parent population include batch size, learning rate, momentum and nesterov acceleration. Specifically, given a chosen optimizer, if applicable, there is a 80% chance using the momentum and 20% not using it. When using the momentum, there is a 50% chance using the nesterov acceleration. The momentum is randomly selected from [0.1,0.9]. The batch size is uniformly chosen from [64,128,256,512]. The learning rate is also randomly selected from a range [ak,bk] depending on the generation k. The upper and lower bounds of the range are annealed over generations starting from the initial range [a0,b0]. For ADAM, β1=0.9 and β2=0.999 are fixed and only the learning rate is randomized. The m-elitist strategy is applied to 60% of the parent population. The probability of mating of the anchor model is 0.25.