A Matter of Framing:The Impact of Linguistic Formalism on Probing Results
Table 5: Best dev score for word-level tasks over 20 epochs (Acc for classification, MSE for regression).
['task *token.ix', 'en 1.71', 'de 2.48', 'task deprel', 'en 0.95', 'de 0.95']
We observe that grammatical tasks score high, while core role labeling lags behind - in line with the findings of Tenney et al. We observe lower scores for German role labeling which we attribute to the lack of training data. Surprisingly, as we show below, this doesn’t prevent the edge probe from learning to locate relevant role-semantic information in mBERT’s layers.