Highway Transformer: Self-Gating Enhanced Self-Attentive Networks
Table 7: Summary of training time costs of 6-layer Transformer-XL.
['[BOLD] model', '[BOLD] time cost (hour)']
Meanwhile, we find that supplementing SDU gates does not increase much of the time cost in comparison with baselines. Instead, the total running time of each experimental setting is quite similar.