Deep RNNs Encode Soft Hierarchical Syntax
Table 2: Results of the dependency arc prediction task. L0–L4 denote the different layers of the model. DP refers to the RNN trained with dependency parsing supervision.
['[BOLD] Source Model', '[BOLD] GloVe', '[BOLD] L0', '[BOLD] L1', '[BOLD] L2', '[BOLD] L3', '[BOLD] L4']
We see the best performance from the dependency parser, finding that the performance for the dependency parser’s representations continue to improve in the deepest layers, with a maximum performance of approximately 95% on the last layer. This result is unsurprising, as this closely related to the task on which the model was explicitly trained. In the three other models, we find peaks that occur 12 to 20 accuracy points above the input layer’s performance. These results support the findings from the constituency label prediction task and show that these findings hold up across syntactic formalisms.