Deep Feed-forward Sequential Memory Networks for Speech Synthesis
Table 1: Comparison of various DFSMN models in objective measures
['ID', 'Network', '#Layers', 'Order', 'Objective Measures MCD', 'Objective Measures F0 RMSE', 'Objective Measures BAPD', 'Objective Measures U/V Error', 'Objective Measures MSE', 'Size (MB)', 'FLOPS(G)']
All DFSMN models are a composition of DFSMN layers and fully-connected layers. One DFSMN layer has 2048 hidden units and 512 projection units, meanwhile each fully-connected layer has 2048 hidden units. Besides, column 4 is in form of N1,N2,s1,s2, indicating that each DFSMN layer looks back N1 frames with stride s1 and looks ahead N2 frames with stride s2. Here we let DFSMN look back and ahead same frames for simplicity. Since the class of FSMN models has not been applied to TTS tasks ever before, we start exploration from a DFSMN model which is shallow and short-sighted. 3 DFSMN layers and the memory order and stride are both 1, which makes it close to a DNN. We then increase the order of DFSMN models successively (1⇒2⇒5⇒10) by fixing the number of DFSMN layers. From D to F, we fix the order and stride as 10,10,2,2 and increase the network depth (3⇒6⇒10). Again, from F, we continue to increase the order of DFSMN models (10⇒20⇒40⇒80), finally reaching a context window of 3200 frames, approaching max length of utterances in the corpus. These objective measures include root mean squared error (RMSE) of F0 in Hz, unvoiced/voiced (U/V) prediction errors, mel-cepstral distortion (MCD) in dB, BAP Distortion (BAPD) and the total MSE on normalized acoustic features. The model size and the FLOPS (floating-point operations to generate per second speech) are also recorded. All objective measures are calculated on the 3 hour validation data.