Memory Transformer
Table 1: Performance of baseline and memory models on WMT-14 DE-EN translation task. Values represent an average of BLEU 4 scores for 3 runs of every model evaluated on a validation set of 2000 segments.
['[BOLD] Small models (4 layers per encoder/decoder) (20 epochs)', '[BOLD] Small models (4 layers per encoder/decoder) (20 epochs)', '[BOLD] Base models (6 layers per encoder/decoder) (10 epochs)']
To verify the training results, we calculated BLEU scores on the validation set to estimate the quality of translation produced by the models. After 20 epochs of training, small MemTransformer models have similar scores and clearly outperform the Transformer baseline. Base 6-layer MemTransformer with 10 memory tokens improves the baseline, and doubling the memory up to 20 tokens results in an even higher score of 25.58. Storing of representations in dedicated memory should be useful not only during encoding but also during decoding. MemTransformer model can be easily run with memory augmented decoder. As the simplest implementation of such memory, we just prepended target sequence with [mem] tokens. Note that we left the Transformer decoder as is, so the model was forced to learn the full target sequence prediction, starting with [mem] tokens first, and then continuing with the tokens for translation. Forward masking was applied to the full input sequence. Indeed, this modification increased difficulty of the learning task, and MemTransformer Enc(0) Dec(10) But adding memory to decoder allowed MemTransformer Enc(10) Dec(10) model slightly improve the baseline scores.