Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling
Table 2: Performances with different model strides on WSJ dataset.
['Loss (stride)', 'WER 2', 'WER 4', 'Epoch Time (mins) 2', 'Epoch Time (mins) 4']
DS2 [Amodei et al., employed non-overlapping bi-gram outputs to allow for a larger stride. This imposes an artificial constraint forcing the model to learn, not only the spelling of each word, but also how to split words into bi-grams. For example, part is split as [pa, rt] but the word apart is forced to be decomposed as [ap, ar, t]. Gram-CTC removes this constraint by allowing the model to decompose words into larger units into the most convenient or sensible decomposition. For Gram-CTC, we use the auto-refined gram set from previous section. As expected, using stride 4 almost cuts the training time per epoch into half, compared to stride 2. From stride 2 to stride 4, the performance of uni-gram CTC drops quickly. This is because small grams inherently need higher time resolutions. As for Gram-CTC, from stride 2 to stride 4, its performance decreases a little bit, while in experiments on the other datasets, Gram-CTC constantly works better in stride 4. One possible explanation is that WSJ is too small for Gram-CTC to learn large grams well. In contrast, the performance of bi-gram CTC is not as good as that of Gram-CTC in either stride.