FlauBERT: Unsupervised Language Model Pre-training for French
Table 6: Constituency parsing and POS tagging results.
['Model', 'Dev F1', 'Dev POS', 'Test F1', 'Test POS']
Without pre-training, we replicate the result from \newcitekitaev-klein-2018-constituency. FastText pre-trained embeddings do not bring improvement over this already strong model. When using pre-trained language models, we observe that CamemBERT, with its language-specific training improves over mBERT by 0.9 absolute F1. FlauBERT\textscbase outperforms CamemBERT by 0.7 absolute F1 on the test set and obtains the best published results on the task for a single model. Regarding POS tagging, all large-scale pre-trained language models obtain similar results (98.1-98.2), and outperform models without pre-training or with fastText embeddings (97.5-97.7). FlauBERT\textsclarge provides a marginal improvement on the development set, and fails to reach FlauBERT\textscbase results on the test set. The ensemble model improves by 0.4 absolute F1 over FlauBERT on the development set and 0.2 on the test set, obtaining the highest result for the task. This result suggests that both pre-trained language models are complementary and have their own strengths and weaknesses.