Accessing Higher-level Representations in Sequential Transformers with Feedback Memory
Table 6: Training Time. The number of days required for training on 32 V100 GPUs. We report perplexity on the validation set.
['[BOLD] Model', '[BOLD] PPL', '[BOLD] Training days']
However, when the temporal parallelization is possible, we compare several different ways of shortening the training time. Further initializing from a pre-trained Transformer reduces the training time to 5.5 days, again without loss in performance.