Køpsala: Transition-Based Graph Parsing via Efficient Training and Effective Encoding
Table 4: Development ELAS for our fixed parser. While in all cases we train the parser on the concatenation of all of a language’s gold treebanks (applicable only to Czech, Dutch, Estonian, and Polish), stanza and udpipe refer to generating predictions on the development data preprocessed by the corresponding pipeline. Gold Tok. refers to generating predictions over gold development data (tokenization, etc).
['[BOLD] Language', 'stanza', 'udpipe', '[BOLD] Gold Tok.']
Since the test data was provided in a raw, untokenized format, we were interested in the extent of accuracy loss we might observe when relying on off-the-shelf pre-processors. It is clear that when we employ predicted segmentation, etc. using either stanza or udpipe pipelines, we observe a slight degradation in accuracy, as compared to the gold data. Omitting Czech, Estonian, Dutch, and Polish (which had several associated treebanks), all other languages degrade by an average of 2.00 ELAS for stanza and 2.32 for udpipe. Though one typically expects such a degradation when evaluating with predicted segmentation, we did observe some unreasonably large gaps in accuracy: namely for Arabic ( −4.02, −8.32 for stanza and udpipe, respectively) and Tamil (−12.19, −8.59). The latter can likely be explained via its small training set, which undoubtedly affects all components of the preprocessing pipeline.