Syntax-aware Multilingual Semantic Role Labeling
Table 3: Semantic F1-score on CoNLL-2009 in-domain test set. The first row is the best result of CoNLL-2009 shared task Hajič et al. (2009). The previously best published results of Catalan and Japanese is from Zhao et al. (2009a), Chinese from Cai et al. (2018), Czech from Marcheggiani et al. (2017), English from Li et al. (2019), German from Roth and Lapata (2016) and Spanish from Kasai et al. (2019).
['Model', 'Catalan', 'Chinese', 'Czech', 'English', 'German', 'Japanese', 'Spanish']
So far, the best previously reported results of Catalan, Japanese and Spanish are still from CoNLL-2009 shared task. Compared with previous methods, our baseline yields strong performance on all datasets except German. Especially for Catalan, Czech, Japanese and Spanish, our baseline performs better than existing methods with a large margin of 3.5% F1 on average. Nevertheless, applying our argument pruning to the strong syntax-agnostic baseline can still boost the model performance, which demonstrates the effectiveness of proposed method. On the other hand, it indicates that syntax is generally beneficial to multiple languages, and can enhance the multilingual SRL performance with effective syntactic integration. The use of contextualized word representation further improves model performance, which overwhelmingly outperforms previously published best results and achieves the new state of the art in multilingual SRL for the first time. Furthermore, we find that ELMo promotes the overall performance of SRL model, but BERT gives more significant performance increase than ELMo on all languages, which suggests that BERT is better at enriching contextual information. More interestingly, we observe that the performance gains from the proposed argument pruning method or these deep enhanced representations are relatively marginal on Japanese, one possible reason is the relatively small size of its training set. This observation also indicates that ELMo or BERT is more suitable for learning on large annotated corpus.