FlauBERT: Unsupervised Language Model Pre-training for French
Table 9: F1 scores (%) on the Noun Disambiguation Task.
['Model', 'Single Mean', 'Single Std', 'Ensemble']
At prediction time, we take the synset ID which has the maximum value along the softmax layer (no filter on the lemma of the target is performed). We trained 8 models for every experiment, and we report the mean results, and the standard deviation of the individual models, and also the result of an ensemble of models, which averages the output of the softmax layer. Finally, we compared FlauBERT with CamemBERT, mBERT, fastText and with no input embeddings. On this task and with these settings, we first observe an advantage for mBERT over both CamemBERT and FlauBERT\textscbase. We think that it might be due to the fact that the training corpora we used are machine translated from English to French, so the multilingual nature of mBERT makes it probably more fitted for the task. Comparing CamemBERT to FlauBERT\textscbase, we see a small improvement in the former model, and we think that this might be due to the difference in the sizes of pre-training corpora. Finally, with our FlauBERT\textsclarge model, we obtain the best scores on the task, achieving more than 1 point above mBERT.