Training Tips for the Transformer Model
Table 5: Maximum batch size which fits into 11GB memory for various combinations of max_length (maximum sentence length in subwords), model size (base or big) and optimizer (Adam or Adafactor). The last two columns show the percentage of sentences in the train (CzEng 1.7) and test (wmt13) data that are longer than a given threshold.
['max_length', 'maximum batch size BIG+Adam', 'maximum batch size BIG+Adafactor', 'maximum batch size BASE+Adam', 'longer sentences train', 'longer sentences test']
The parameter max_length specifies the maximum length of a sentence in subwords. Longer sentences (either in source or target language) are excluded from the training completely. If no max_length is specified (which is the default), batch_size is used instead. Lowering the max_length allows to use a higher batch size or a bigger model. Since the Transformer implementation in T2T can suddenly run out of memory even after several hours of training, it is good to know how large batch size fits in your GPU. Setting max_length too low would result in excluding too many training sentences and biasing the translation towards shorter sentences, which would hurt the translation quality. Even with max_length 150 or 200 the BLEU learning curve is worse than with max_length=400, which finally gives the same result as not using any max_length restriction. We have no explanation for this phenomenon.