TinyBERT: Distilling BERT for Natural Language Understanding
Table 7: Results (dev) of different mapping strategies.
['System', 'MNLI-m', 'MNLI-mm', 'MRPC', 'CoLA', 'Average']
We find that the top-strategy performs better than the bottom-strategy in MNLI, while being worse in MRPC and CoLA tasks, which confirms the observations that different tasks depend on the different kinds of knowledge from BERT layers. Since the uniform-strategy acquires the knowledge from bottom to top layers of BERTBASE, it achieves better performances than the other two baselines in all the four tasks. Adaptively choosing layers for a specific task is a challenging problem and we leave it as the future work.