From POS tagging to dependency parsing for biomedical event extraction
Table 4: Parsing results on the test set with predicted POS tags and gold tokenization (except [G] which denotes results when employing gold POS tags in both training and testing phases). “Without punctuation” refers to results excluding punctuation and other symbols from evaluation. “Exact match” denotes the percentage of sentences whose predicted trees are entirely correct [25]. [∙] denotes the use of the pre-trained Stanford tagger for predicting POS tags on test set, instead of using the retrained NLP4J-POS model. Score differences between the “retrained” parsers on both corpora are significant at p≤0.001 using McNemar’s test (except UAS scores obtained by Stanford-Biaffine-v2 for gold and predicted POS tags on GENIA, i.e. 92.51 vs. 92.31 and 92.84 vs. 92.64, where p≤0.05).
['[BOLD] System', '[BOLD] System', '[BOLD] System', '[BOLD] With punctuation  [BOLD] Overall', '[BOLD] With punctuation  [BOLD] Overall', '[BOLD] With punctuation  [BOLD] Exact match', '[BOLD] With punctuation  [BOLD] Exact match', '[BOLD] Without punctuation  [BOLD] Overall', '[BOLD] Without punctuation  [BOLD] Overall', '[BOLD] Without punctuation  [BOLD] Exact match', '[BOLD] Without punctuation  [BOLD] Exact match']
In this section, we present our empirical approach to evaluate different POS tagging and dependency parsing models on benchmark biomedical corpora. The first five rows show parsing results on the GENIA test set of “pre-trained” parsers. Both pre-trained NNdep and Biaffine models were trained on a dependency treebank of 40K sentences, which was converted from the English PTB sections 2–21. The remaining rows show results of our retrained dependency parsing models. Regarding the retrained parsing models, on both GENIA and CRAFT, Stanford-Biaffine achieves the highest parsing results with LAS at 91.23% and UAS at 92.64% on GENIA, and LAS at 90.77% and UAS at 92.67% on CRAFT, computed without punctuations. Stanford-NNdep obtains the lowest scores; about 3.5% and 5% absolute lower than Stanford-Biaffine on GENIA and CRAFT, respectively. jPTDP is ranked second, obtaining about 1% and 2% lower scores than Stanford-Biaffine and 1.5% and 1% higher scores (without punctuation) than NLP4J-dep on GENIA and CRAFT, respectively. the intrinsic UAS and LAS (F1) scores on the pre-processed segmented BioNLP 2009 development sentences (i.e. scores with respect to predicted segmentation), for which these sentences contain event interactions. Although gold event annotations in the BioNLP 2009 test set are not available to public, it is likely that we would obtain the similar intrinsic UAS and LAS scores on the pre-processed segmented test sentences containing event interactions. This is logical because the BioNLP 2009 shared task dataset was a subset of the GENIA corpus. Among the four dependency parsers trained on GENIA, Stanford-Biaffine, jPTDP and NLP4J-dep produce similar event extraction scores on the development set, while on the the test set jPTDP and NLP4J-dep obtain the lowest and highest scores, respectively.