Highway Transformer: Self-Gating Enhanced Self-Attentive Networks
Table 4: The performance of 3-layer R-Transformers (RT-L3) and SDU components on word-level PTB LM.
['model', 'eval loss', 'eval ppl', 'test loss', 'test ppl']
As in Fig. Similar to the word-level R-Transformer, tanh-gate SDUs behave a bit better than the counterpart with sigmoid gates, both showing stable curvatures of convergence.