Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
Table 3: Unlabeled parsing F1 results evaluated on WSJ10 and PTB23.
['[BOLD] Model', '[BOLD] Parsing F1  [BOLD] WSJ10', '[BOLD] Parsing F1  [BOLD] PTB23', '[BOLD] Accuracy on PTB23 by Tag  [BOLD] NP', '[BOLD] Accuracy on PTB23 by Tag  [BOLD] VP', '[BOLD] Accuracy on PTB23 by Tag  [BOLD] PP', '[BOLD] Accuracy on PTB23 by Tag  [BOLD] S', '[BOLD] Accuracy on PTB23 by Tag  [BOLD] SBAR']
From the table, we see that BERT outperforms most baselines on PTB23, except for the second layer of ON-LSTM. Note that all these baselines have specifically-designed architectures for the unsupervised parsing task, while BERT’s knowledge about constituent formalism emerges purely from self-supervised training on unlabeled text. To further understand the strengths and weaknesses of each system, we analyze their accuracies by constituent tags. We find that the success of PRPN and ON-LSTM mainly comes from the accurate identification of NP (noun phrase), which accounts for 38.5% of all constituents. For other phrase-level tags like VP (verb phrase) and PP (prepositional phrase), the accuracies of BERT are competitive. Moreover, for clause level tags, BERT significantly outplays ON-LSTM. Take SBAR (clause introduced by a subordinating conjunction) for example, BERT achieves an accuracy of 51.9%, which is about 3.4 times higher than that of ON-LSTM. One possible interpretation is that BERT is pre-trained on long contiguous sequences extracted from a document-level corpus. And the masking strategy (randomly mask 15% tokens) utilized may allow BERT to learn to model a sequence of words (might form a clause).