Adversarial Transfer Learning for Punctuation Restoration
TABLE V: Transferring parameters from pre-trained BERT to punctuation predicting models. The results of punctuation predicting models in terms of P(%) ,R(%) , F1(%) on test sets of IWSLT2011 datasets.
['Test', 'Model', 'Initialization', '[ITALIC] COMMA  [ITALIC] P', '[ITALIC] COMMA  [ITALIC] R', '[ITALIC] COMMA  [ITALIC] F1', '[ITALIC] PERIOD  [ITALIC] P', '[ITALIC] PERIOD  [ITALIC] R', '[ITALIC] PERIOD  [ITALIC] F1', '[ITALIC] QUESTION  [ITALIC] P', '[ITALIC] QUESTION  [ITALIC] R', '[ITALIC] QUESTION  [ITALIC] F1', '[ITALIC] Overall  [ITALIC] P', '[ITALIC] Overall  [ITALIC] R', '[ITALIC] Overall  [ITALIC] F1']
The results show that the BERT-BLSTM-CRF models obtain better results than the BERT-CRF models accordingly. The BERT-CRF models with random initialization achieve the worst results among our models on both two test sets. But the BERT-BLSTM-CRF models with transferred parameters from pre-trained BERT model obtain the best performance on both two test sets: Ref. and ASR.