Character-based NMT with Transformer
Table 4: Results for the high resource setting.
['enc', 'Char PreN', 'Char PreN + T', '5\u2009000 PreN', '5\u2009000 PreN+T', '30\u2009000 PreN', '30\u2009000 PreN+T']
, we no longer train models with post layer normalization and restrict ourselves to pre layer normalization and transparent attention. Here, we also experiment with the BPE 30 000 vocabulary. Here again we see an improvement in BLEU scores with increasing depth of 1-2 points when going beyond 6 encoder layers. Transparent attention seems to help consistently for character level models but barely does anything for the two BPE models. Further, with increased depth, BPE 5 000 and 30 000 perform similarly in contrast to shallow models where there is a 1 BLEU difference. However, character level models are still slightly worse than the BPE models with a max score of 37.7 rather than 38 for the BPE models.