Character-based NMT with Transformer
Table 3: Results for the low resource setting. “PreN + T” refers to an architecture with layer normalization before each sub-layer and transparent attention. Best results for each layer depth are shown in bold.
['enc', 'Char PreN', 'Char PreN+T', '5\u2009000 PreN', '5\u2009000 PreN+T']
While adding transparent attention is beneficial for almost all depths in the character level models, it gives mixed results for the BPE 5 000 model. Hence, we see that by training deeper models we are able to marginally improve performance for both vocabularies. For character level models we improve by 1 BLEU points (from 33.7 to 34.7). For 5 000, the gain is a more modest 0.4 BLEU points (from 35 to 35.4). We are able to narrow but not close the gap between character and BPE.