Training Tips for the Transformer Model
Table 7: WMT17 systems for English-to-Czech and our best T2T training run. Manual scores are from the official WMT17 ranking. Automatic metrics were provided by http://matrix.statmt.org/. For *TER metrics, lower is better. Best results in bold, second-best in italics.
['Manual #', 'Manual Ave %', 'Manual Ave z', 'Automatic Scores BLEU', 'Automatic Scores TER', 'Automatic Scores CharacTER', 'Automatic Scores BEER', 'System']
While the automatic evaluation is not fully reliable (see e.g. the high BLEU score for CU-Chimera despite its lower manual rank), we see that the Transformer model outperforms the best system in BLEU, TER, CharacTER and BEER, despite it does not use any back-translated data, reranking with other models (e.g. right-to-left reranking) nor ensembling (as is the case of uedin-nmt and other systems). Note that our Transformer uses a subset of the constrained training data for WMT17, so the results are comparable.