Deepening Hidden Representations from Pre-trained Language Models
Table 5: Effect of dynamic mechanism when computing the importance scores. A median Matthews correlation of five random runs is reported for CoLA on the development set.
['Method', 'Dev set CoLA', '[EMPTY]']
We investigate how the adaptive assignment mechanism of the importance score affects the model’s performance. CoLA is chosen as the downstream task. first 6, last 6 or all 25 layers. Two strategies are studied: mean and random. In the mean situation, we suppose all the layers contribute exactly the same. On the contrary, under the random condition, we generate a score for each layer randomly and do the SoftMax operation across all the layers.