Sub-band Knowledge Distillation Framework for Speech Enhancement
Table 2: The mean square error (MSE, ℓ2 loss) of the teacher models and the student models (without the guidance of the teacher models) with the different number of memory cells per layer in LSTM for each sub-band on the test set. C is the number of memory cells per layer in LSTM.
['C', 'Student Model 0-40', 'Student Model 40-80 (10−4)', 'Student Model 80-120 (10−4)', 'Student Model 120-160 (10−4)', 'Teacher Model 0-40', 'Teacher Model 40-80 (10−4)', 'Teacher Model 80-120 (10−4)', 'Teacher Model 120-160 (10−4)']
C is the number of memory cells. The left half of the table is the result area of the student models, and the right half is the result area of the teacher models. For example, in the result area of the student models, the value in the first column (0 to 40) of the first row (C is 256) shows the result of the student model with 256 memory cells and for the 0 to 40 sub-band. The input to the student model is one of the four sub-bands in the training sample (random reselection for each iteration). The training target of the student model is solely the corresponding sub-band of the clean speech. The value of 0.036 represents the ℓ2 loss between the enhanced test sample and the target. In the result area of the teacher models, the first column (0 to 40) of the first row (C is 256) lists the teacher model result with 64 memory cells and for the 0 to 40 sub-band. The training data is a sub-band with a frequency range of 0 to 40 in each sample, and the target is the corresponding sub-band of the clean speech. The value of 0.031 is the ℓ2 loss between the enhanced test sample and the target. 10−4 in the table means that the value in the table is multiplied by 10−4. In total, we trained 3 student models (3 different numbers of memory cells) and 12 teacher models (3 different numbers of memory cells × 4 sub-bands).