Multi-Scale Self-Attention for Text Classification
Table 5: Test Accuracy on SNLI dataset for sentence vector-based models.
['Model', 'Acc']
The reported number of Transformer is obtained with heuristic hyper-parameter selection, we use a three-layer Transformer with heavy dropout and weight decay. And there is still a large margin compared to Multi-Scale Transformer. This comparison also indicates the moderate size training data (SNLI has 550k training samples) cannot replace the usefulness of prior knowledge.