Multi-Task Self-Supervised Learning for Disfluency Detection
Table 5: Ablation over model size. #L = the number of layers; #H = hidden size; #A = number of attention heads. “Full” means fine-tuning on 100% gold training data, and “1000 sents” means fine-tuning on less than 1% (1000 sentences) gold training data.
['Hyperparams #L', 'Hyperparams #H', 'Hyperparams #A', 'Dev Set Performance F1 (1000 sents)', 'Dev Set Performance F1 (Full)']
We explore the effect of model size on fine-tuning disfluency detection task. We mainly tune the number of layers, hidden units, and attention heads, while otherwise using the same hyperparameters as described previously. In order to avoid the impact of randomness, we run 3 random restarts of fine-tuning and report the average F1-Score on the Dev set. We can see that larger models always lead to a strict F1-score improvement on the Dev set. It is also surprising that larger models always reduce the performance gap between fine-tuning using full gold training data and fine-tuning using less than 1% (1000 sentences) training data. We believe that much more performance will be gained if we keep increasing the model size.