Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling
Table 1: Results of different gram selection methods on WSJ dataset.
['Loss', 'WER']
In our experiments, we first eliminate most of useless grams from the statistics of a huge corpus, that is, we count the frequency of each gram in the corpus and drop these grams with rare frequencies. Then, we train a model with Gram-CTC on all the remaining grams. By applying (decoding) the trained model on a large speech dataset, we get the real statistics of gram’s usage. Ultimately, we choose high frequency grams together with all uni-grams as our final gram set We first train a model using all uni-grams and bi-grams (29 uni-grams and 262=676 bi-grams, in total 705 grams), and then do decoding with the obtained model on another speech dataset to get the statistics of the usage of grams. Top 100 bi-grams together with all 29 uni-grams (auto-refined grams) are used for the second round of training. For comparison, we also present the result of the best hand-picked grams, as well as the results on uni-grams. First, the performance of auto-refined grams is only slightly better than the combination of all uni-grams and all bi-grams. This is probably because WSJ is so small that gram learning suffers from the data sparsity problem here (similar to word-segmented models). The auto-refined gram set contains only a small subset of bi-grams, thus more robust. This is also why we only try bi-grams, not including higher-order grams. Second, the performance of best handpicked grams is worse than auto-refined grams. This is desirable. It is time-consuming to handpick grams, especially when you consider high-order grams. The method of iterative gram selection is not only fast, but usually better. Third, the performance of Gram-CTC on auto-refined grams is only slightly better than CTC on uni-grams. This is because Gram-CTC is inherently difficult to train, since it needs to learn both decomposition and alignment. WSJ is too small to provide enough data to train Gram-CTC.