Modeling Recurrence for Transformer
Table 4: Comparison with re-implemented related work: “RelPos”: relative position encoding Shaw et al. (2018), “DiSAN”: directional SAN Shen et al. (2018), “RNN Encoder”: combining SAN and RNN encoders with multi-column strategy Chen et al. (2018).
['[BOLD] Model', '[BOLD] BLEU']
As seen, all the recurrence enhanced approaches achieve improvements over the baseline model Transformer-base, which demonstrates the necessity of modeling recurrence for Transformer. Among these approaches, our approach (i.e., 1-Layer BiARN Encoder) achieves the best performance.