Multi-Task Self-Supervised Learning for Disfluency Detection
Table 7: Comparison with BERT. “random-initial” means training transformer network on gold disfluency detection data with random initialization. “combine” means concatenating hidden representations of BERT and our self-supervised models for fine-tuning.
['[BOLD] Method', '[BOLD] F1 (Full)', '[BOLD] F1 (1000 sents)']
Compared with BERT, we use much smaller training corpus and model parameters (6-layer transformer blocks, 512 hidden-size, and 8 self-attention heads) limited by devices. Both our method and BERT outperforms the baseline model with random initialization, which proves the strong ability of pre-training model. Although our pre-training corpus and model parameters are much smaller than BERT, we achieve a similar result with BERT when fine-tuning on full gold training data. Surprisingly, our method achieves 3.7 point improvements over BERT when fine-tuning on 1%(1000 sentences) of the gold training data. We also try to combine our pre-train model and BERT by concatenating their hidden representation. The result shows that much higher performance is achieved. The performance is always much higher by combining our pre-train model and BERT. This proves that our model and BERT can coexist harmoniously, and capture different aspects of information helpful for disfluency detection.