SemEval 2019 Shared Task: Cross-lingual Semantic Parsing with UCCA Call for Participation
Table 1: Pilot task results, in percents, on v1.2 of the English Wiki test set (left) and v1.0 of the 20K Leagues set (right). Columns correspond to labeled precision, recall and F1, for both primary and remote edges. F1 upper bounds are reported for the conversions. TUPASparse and TUPAMLP are from Hershcovich et al. (2017), as well as the conversion-based baselines: DAGParser Ribeyre et al. (2014), TurboParser Almeida and Martins (2015), uparse Maier and Lichte (2016), MaltParser Nivre et al. (2007), and the LSTM parser Dyer et al. (2015). TUPADense and TUPABiLSTM PoE are unpublished results. TUPABiLSTM and TUPABiLSTM MTL are from Hershcovich et al. (2018).
['[EMPTY]', 'Wiki (in-domain) Primary', 'Wiki (in-domain) Primary', 'Wiki (in-domain) Primary', 'Wiki (in-domain) Remote', 'Wiki (in-domain) Remote', 'Wiki (in-domain) Remote', '20K Leagues (out-of-domain) Primary', '20K Leagues (out-of-domain) Primary', '20K Leagues (out-of-domain) Primary', '20K Leagues (out-of-domain) Remote', '20K Leagues (out-of-domain) Remote', '20K Leagues (out-of-domain) Remote']
It was carried out on version 1.2 of the Wiki dataset, and version 1.0 of the English 20K Leagues dataset. An unpublished experiment using a perceptron with randomly-initialized embedding inputs, instead of sparse features, yielded poor results (TUPADense). Another unpublished experiment with an ensemble of three BiLSTM models (TUPABiLSTM PoE) yielded the best results on primary edges in the in-domain setting (75% F1), but no improvement on remote edges (48.7% F1). An improvement was also observed in the out-of-domain setting, obtaining 69.6% F1 on primary edges and 28% F1 on remote edges.