Distilling Neural Networks for Greener and Faster Dependency Parsing
Table 6: Speeds with batch size 256.
['[BOLD] gr', 'CPU', '(tok/s)', '[BOLD] Full 1211', '[BOLD] Full ±', '[BOLD] Full 2', '[BOLD] D-20 2842', '[BOLD] D-20 ±', '[BOLD] D-20 3', '[BOLD] D-40 2086', '[BOLD] D-40 ±', '[BOLD] D-40 3', '[BOLD] D-60 1638', '[BOLD] D-60 ±', '[BOLD] D-60 3', '[BOLD] D-80 1390', '[BOLD] D-80 ±', '[BOLD] D-80 1']
As expected, a much smaller batch size is required to achieve increases in parsing speed when using a CPU. Even with a batch size of 16 sentences, the smallest model more than doubles the speed of the baseline. For a batch size of 256, the distilled model compressed to 20% increases the speed of the baseline by 130% when considering tokens per second.