Deepening Hidden Representations from Pre-trained Language Models
Table 4: Ablation study over model design consideration on the development set of CoLA and STS-B. The result for each model is a median of five random runs. Both HIRE and fusion network significantly improve the model performance on all two datasets.
['Model', 'Dev set CoLA', 'Dev set STS-B']
To individually evaluate the importance of HIRE and fusion network, we vary our model in the following ways and conduct the experiments on the development set of CoLA and STS-B. In order to keep the fusion network, we fuse the preliminary representation with itself, which means we define instead M by: M=[R;R;R+R;R∘R]∈Rn×4d (12) The results are presented in row 2. We remove the fusion network, and take the outputs of HIRE as the final representation of the input and flow it directly into the output layer and present the results in row 3. As can be seen from the table, to use of HIRE to extract the hidden information is crucial to the model: Matthews correlation of CoLA and Pearson correlation coefficient of STS-B drop dramatically by 1.2 and 0.9 points if it’s removed. We also observe that fusion network is an important component that contributes 1.5/0.2 gains on the CoLA and STS-B tasks. The results are aligned with our assumption that HIRE extracts the complementary information from the hidden layers while the fusion network fuses it with the preliminary representation.