Multimodal Semi-supervised Learning Framework for Punctuation Prediction in Conversational Speech
Table 3: F1 scores for punctuation prediction (streaming).
['Model', 'NP', 'Comma', 'FS', 'QM']
We have conducted experiments to study the real time (streaming) performance of the proposed model when there is no future context. For the purpose of this experiment, we perform upper triangle masking (similar to transformer [vaswani2017attention] decoder layers) in self attention layers of lexical encoder to mask the right side context. Since the acoustic encoder is unidirectional, we did not make any further changes. as acoustic features. We also trained an additional 4 layer LSTM model for baseline comparison. Similar to bidirectional models, the pretrained BERT model performs ∼2% better than LSTM model thus proving that pretraining also helps in learning better representations in the absence of right side context. We also observe that adding acoustic information leads to an additional ∼3% improvement over the lexical BERT model, confirming the effectiveness of our proposed approach for streaming usecase.