Reducing Transformer Depth on Demand with Structured Dropout
Table 6: BLEU for IWSLT (test set).
['Model', 'BLEU']
We see small improvement, likely as the network is small and already has a large quantity of regularization with dropout, attention dropout, and weight decay. The Transformer is not the state of the art architecture, and there remains a large gap between the Transformer and the DynamicConv model proposed by Wu et al.