DiscreTalk: Text-to-Speech as a Machine Translation Problem
Table 3: Effectiveness of shallow fusion with the VQ-LM.
['LM weight', 'CER [%]', 'TER [%]']
Finally, we investigated the performance of shallow fusion with the LM for the subword sequence of the discrete symbols (VQ-LM). We built three long short-term memory (LSTM)-based VQ-LMs with 1,024 units and a different number of layers (1, 2, and 4) for the subword sequence (DSF128, SW256). The training curve showed the same tendency as the NMT models, which tended to overfit in the early stage, and the deeper model brought lower training perplexity (10.6, 7.6, and 5.3, respectively). To check the effectiveness of the fusion, we used the best perplexity VQ-LM model and changed the weight for its score in decoding from 0.1 to 0.3 while fixing beam-size to 3. From the results, we could not confirm the improvement. One of the reasons is that the training data was the same for NMT models and VQ-LMs, and the amount of training data was relatively small while the length of each sequence was long. We need to investigate the case where the training data of VQ-LM is much bigger than NMT models; in other words, we have many untranscribed utterances.