Predicting Prosodic Prominence from Text with Pre-trained Contextualized Word Representations
Table 5: Typical 3-way prominence predictions of BERT compared to reference labels.
['REF: BERT:', 'In', 'the', '[BOLD] next', 'moment', 'he', 'was', '[BOLD] concealed', 'by', 'the', 'leaves', '.']
Overall, our results for BERT highlight the importance of pre-training of the word representations. As we noticed, already with as little as 10% of the training data, BERT outperforms the other models when they are trained on the entire training set. This suggests that BERT has implicitly learned syntactic or semantic information relevant for the prosody prediction task. Our results are in line with the earlier results by Stehwien et al. et al. These examples indicate that even if the overall accuracy of the model is not high, the predictions still look plausible in isolation.