Køpsala: Transition-Based Graph Parsing via Efficient Training and Effective Encoding
Table 3: Main results for Enhanced Universal Dependencies shared task (ELAS), as evaluated on the provided test sets. B1, B2, B3 refer to organizer-provided baseline systems. official refers to our official submission, prior to fixing the unconnected graph issue (fixed).
['[BOLD] Language', '[BOLD] Baselines  [ITALIC] B1', '[BOLD] Baselines  [ITALIC] B2', '[BOLD] Baselines  [ITALIC] B3', '[BOLD] Ours  [ITALIC] official', '[BOLD] Ours  [ITALIC] fixed']
Though we observed scant differences between the two preprocessing pipelines, it was udpipe that produced fewer validation errors. As such, we adopted it as the main preprocessor for our official submission. After diagnosing and fixing this problem, we observed an improvement of 14.1 ELAS, which is consistent with our scores on the treebanks’ development sets. With this in mind, our (fixed) parser tends to perform in a generally stable fashion across languages, with an average ELAS of 76.48 and standard deviation of 6.86. Among our highest scoring languages are Bulgarian, French, and Italian— the former two of which are corroborated by the strong B1 baseline. Indeed, Tamil is the notable exception among all results, with 56.85 ELAS. We surmise that treebank size is the biggest factor in this degradation of performance, as Tamil has, by far, the smallest treebank at 400 sentences. As such, our parser has comparatively fewer graph samples to train on than it would for some other languages.