Highway Transformer: Self-Gating Enhanced Self-Attentive Networks
Table 1: Performance of 3 Layer Transformers and SDU components on char-level PTB LM task. The best performance is marked bold.
['model', 'eval loss', 'eval ppl', 'test loss', 'test ppl']
Furthermore, Fig. In which SDUs with tanh gates (8.76% improvement) outperform the counterpart with sigmoid gates (8.2% improvement) in terms of the final perplexity on the test set.