Understanding and Improving TransformerFrom a Multi-Particle Dynamic System Point of View
Table 2: Test results on the GLUE benchmark (except WNLI).
['[BOLD] Method', 'CoLA', 'SST-2', 'MRPC', 'STS-B', 'QQP', 'MNLI-m/mm', 'QNLI', 'RTE', '[BOLD] GLUE']
We present the results of two BERT base models. We can see from the table, our proposed Macaron Net base outperforms all baselines in terms of the general GLUE score. Specifically, given the same dataset, our proposed model outperforms our trained BERT base model in all tasks. Comparing our results with original BERT model, we can see that given a fixed number of parameters and a fixed number of attention sub-layers, stacking more FFN sub-layers can get better performance on downstream tasks.