Distilling Neural Networks for Greener and Faster Dependency Parsing
Table 7: Hyperparameters for full-sized baseline models.
['[BOLD] hyperparameter', '[BOLD] value']
We train Biaffine parsers and apply the teacher-student distillation method to compress these models into a number of different sizes for a number of Universal Treebanks v2.4 (UD) During distillation dropout is not used as in earlier experiments with dropout performance was hampered. And subsequent work on distillation which uses dropout also didn’t perform well, but it isn’t clear if this is the cause of the poorer performance, e.g. different treebanks were used, UPOS tags weren’t, and no pre-trained embeddings were used Dehouck et al. Beyond lexical features, the model only utilises universal part-of-speech (UPOS) tags. Gold UPOS tags were used for training and at runtime. Also, we used gold sentence segmentation and tokenisation. We opted to use these settings to compare models under homogeneous settings, so as to make reproducibility of and comparability with our results easier.