Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
Table 4: Performance of different discourse parser. The distance is defined as the number of EDUs between head and dependent.
['[BOLD] Model', '[BOLD] UAS', '[BOLD] Accuracy by distance 0', '[BOLD] Accuracy by distance 1', '[BOLD] Accuracy by distance 2', '[BOLD] Accuracy by distance 5']
We find that both Eisner and CLE achieve significantly higher UAS (+28) than the random BERT baseline. This suggests that BERT is aware of the structure of the document it is given. In particular, we observe a decent accuracy in identifying discourse relations between adjacent EDUs, perhaps due to the “next sentence prediction” task in pre-training, as pointed out in shi2019next. Our finding sheds some lights on BERT’s success in downstream tasks that have paragraphs as input (e.g., Question Answering).