Distilling Neural Networks for Greener and Faster Dependency Parsing
Table 3: Full attachment scores for each model and for each test treebank where Full means the original sized model, B-X means training a model with X% of the trainable parameters of the original model, and D-X means distilling to a model with X% of the trainable parameters of the original model.
['[EMPTY]', '[BOLD] gr UAS', '[BOLD] gr LAS', '[BOLD] zh UAS', '[BOLD] zh LAS', '[BOLD] en UAS', '[BOLD] en LAS', '[BOLD] fi UAS', '[BOLD] fi LAS', '[BOLD] he UAS', '[BOLD] he LAS', '[BOLD] ru UAS', '[BOLD] ru LAS', '[BOLD] ta UAS', '[BOLD] ta LAS', '[BOLD] ug UAS', '[BOLD] ug LAS', '[BOLD] wo UAS', '[BOLD] wo LAS', '[BOLD] avg UAS', '[BOLD] avg LAS']
These observations are echoed in the results for the model distilled to 80%, where most treebanks lose less than a point for UAS and LAS against the full baseline, but have a smaller increase in performance over the equivalent-sized baseline. This makes sense as the model is still close in size to the full baseline and still similarly powerful. The increase in performance for Tamil-TTB and Wolof-WTB are greater for this distilled model, which suggests the full model doesn’t need to be compressed to such a small model to help with generalisation.