From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?
Table 1: F-Scores for Hebrew Multi-tagging on the standard dev-set of the Modern Hebrew treebank. +Char means a character-based LSTM encoding, +FT means morphologically-trained Fast-text embeddings. BERT means fine-tuning the contextualized embeddings of Multilingual BERT mlbert. COPYNET is the model we propose in Section 6. Oracle Segmentation means that the segmentation into morphemes (expert annotation) is known in advance. Predicted Segmentation means the decomposition into morpheme automatically predicted via more19. Raw Tokens means that raw input tokens are provided as is, Raw Lattices means that the tokens are automatically transformed into complete morphological lattices based on a wide-coverage symbolic lexicon.
['Model →', 'Pre-Neural SOTA', 'LSTM-CRF', 'LSTM-CRF +Char', 'LSTM-CRF +FT', 'LSTM-CRF +Char+FT', 'Seq2Seq COPYNET', 'BERT']
The pre-neural model obtains 95.5 F1 on joint Seg+POS prediction on the standard dev set. As for the neural models, in an oracle segmentation scenario, where the gold morphological segmentation is known in advance, both BERT and the LSTM-CRF get close to the pre-neural model results. However, they solve an easier and unrealistic task, since in realistic scenarios the gold segmentation is never known in advance. In the more realistic scenarios, where the segmentation is automatically predicted (via \newcitemore19), the results of the Neural models substantially drop. As expected, morph-based and char-based representations help to improve results of the LSTM-CRF model, though not yet reaching the 95 F-score of the pre-neural model. Finally, employing our COPYNET with symbolic morphological lattices, with OOV segmentation heuristics as in the pre-neural model, leads to the most significant improvement, almost closing the gap with the pre-neural state-of-the-art result. Unfortunately, lattices are incompatible with LSTMs and with BERT, since LSTMs and BERT models assume complete linear ordering of the tokens, while lattices impose only a partial order on the morphemes. The question how to incorporate contextualized embeddings into joint, lattice-based, models is fascinating, and calls for further research.