Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
Table 2: ELECTRA pretraining results at the base scale.
['Model size', 'CoLA', 'SST-2', 'MRPC', 'STS-B', 'QQP', 'MNLI', 'QNLI', 'RTE', 'GLUE-AVG']
To further test generality of F-TFM, we additionally consider ELECTRA for pretraining. Overall, we see a similar trend, though the gain is slightly smaller on the GLUE benchmark. This could be attributed to reusing two key hyper-parameters (discriminator loss coefficient and generator size multiplier) tuned for Transformer to train F-TFMs without any adjustment at all.