FlauBERT: Unsupervised Language Model Pre-training for French
Table 8: F1 scores (%) on the Verb Disambiguation Task.
['Model', '[ITALIC] F1']
First we compute sense vector representations from examples found in the Wiktionary sense inventory: given a sense s and its corresponding examples, we compute the vector representation of s by averaging the vector representations of its examples. Then, we tag each test instance with the sense whose representation is the closest based on cosine similarity. We used the contextual embeddings output by FlauBERT as vector representations for any given instance (from the sense inventory or the test data) of a target word. We proceeded the same way with mBERT and CamemBERT for comparison. We also compared our model with a simpler context vector representation called averaged word embeddings (AWE) which consists in representing context of target word by averaging its surrounding words in a given window size. We experimented AWE using fastText word embeddings with a window of size 5. BERT-based models set the new state of the art on this task, with the best results achieved by CamemBERT and FlauBERT\textsclarge.