ShanghaiTech at MRP 2019: Sequence-to-Graph Transduction with Second-Order Edge Inference for Cross-Framework Meaning Representation Parsing
Table 7: Comparing Labeled F1 scores of models with different types of embedding combinations on the development set of the gold DM dataset. Baseline represents the parser of wang-etal-2019-second. Base represents the pre-trained BERT-Base uncased model and Large represents the pre-trained BERT-Large uncased model. fixed and tuned represents whether to fine-tune the BERT model. BERT in the last block represents the last embedding combination (Large-fixed + Glove + Lemma + Char) in the first block. First represents first subtoken pooling, Avg represents average pooling over subtokens. dep-tree represents adding dependency information into embeddings. For each case, we report the highest Labeled F1 score on the development set in our experiments.
['[EMPTY]', 'LF1']
We use BERT devlin-etal-2019-bert embedding in our model. We compared the performance of DM in the original SDP dataset with different subtoken pooling methods, and we also explored whether combining other embeddings such as pre-trained word embedding Glove pennington2014glove and contextual embedding ELMo peters-etal-2018-deep will further improve the performance. We found that Glove, lemma and character embeddings are helpful for DM and fine-tuning on the training set slightly improves the performance. ELMo embedding is also helpful but cannot outperform BERT embedding. However, the performance dropped when ELMo embedding and BERT embedding are combined. We speculate that the drop is caused by the conflict between the two types of contextual information. For subtoken pooling, we compared the performance of using first subtoken pooling and average pooling as token embedding. We found that average pooling is slightly better than first pooling. For syntactic information, we encode each head word and dependency label as embeddings and concatenate them together with other embeddings. The result shows that syntactic information as embeddings is not very helpful for the task. We will try other methods utilizing syntactic information in future work.