Instructions for EMNLP-IJCNLP 2019 Proceedings
Table 3: Effect of the number of update steps and the inner learning rate α.
['[BOLD] Model', '#Upt', '[ITALIC] α', 'CoLA', 'MRPC', 'STS-B', 'RTE']
In this part, we test the effect of the number of update steps k and the learning rate in the inner learning loop. We find that setting k to 5 is the optimal strategy and more or fewer update steps may lead to worse performance. We also vary the inner learning rate α and investigate its impact. We can see that larger α may degrade the performance because the resulting gradients deviate a lot from normal ones. The above two ablations studies demonstrate the importance of making the meta-gradient informative.