Multimodal Semi-supervised Learning Framework for Punctuation Prediction in Conversational Speech
Table 5: Comparison of F1 scores for punctuation with models trained on reference transcripts and ASR augmented data.
['Model', 'n-best', 'NP', 'Comma', 'FS', 'QM']
We have seen that models trained on reference transcripts did not perform that well when tested on ASR outputs. To make models more robust against ASR errors, we perform data augmentation with ASR outputs for training [sunkara2020robust]. For punctuation restoration, we use edit distance measure to align ASR hypothesis with reference punctuated text and restore the punctuation from each word in reference transcription to hypothesis. If there are words that are punctuated in reference but got deleted in ASR hypothesis, we restore the punctuation to previous word. From the results, it is evident that the models trained purely on reference transcripts were outperformed by models trained on augmented text (both reference and ASR outputs). Overall, data augmentation with 3-best lists gave the best performance. Question mark improved by 6% in F1 score by performing data augmentation. The improvement might be due to increased number of training examples in augmented data.