Acknowledgements
Table 5.15: Experiment comparing various stages of parent model and learning rate. Each row corresponds to the same parent model as saved at given step (parent). Columns correspond to different learning rate shifts named by the global step at which it starts. The “0k-child” row is therefore baseline trained only on the child training set. The column “Parent” represent a performance of the parent at the step when the child was spawned.
['Learning rate at: 0k 0k-child', 'Learning rate at: 0k 8 .16', '50k 7 .55', '100k 7 .24', '200k 6 .43', '400k 6 .09', '800k 5 .45', '1600k 4 .61', 'Parent 0.0']
Thus they are comparable, and we present them in the form of a heat-map for better visualization. However, the values within columns should be compared with care as they differ in the parent model that has been trained for a various number of steps. Note that the learning rate scheme of the parent model never changed. has the best performance with learning rate starting at 0k. Therefore the initial warm-up steps (16k in total) are needed for proper training. In contrast, resetting learning rate to 0k across all the transfer setups (see the column “0k”) harms the final child performance. BiRNN Bidirectional Recurrent Neural Network BP Brevity Penalty BPE Byte Pair Encoding CNN Convolutional Neural Network IWSLT International Workshop on Spoken Language Translation LanideNN Language Identification by Neural Networks LM Language Model LSTM Long Short-Term Memory cells MT Machine Translation NLP Natural Language Processing NMT Neural Machine Translation NN Neural Network PBMT Phrase-Based Machine Translation POS Part-of-Speech OOV Out-of-Vocabulary RNN Recurrent Neural Network SMT Statistical Machine Translation T2T Tensor2tensor WAT Workshop on Asian Translation WMT Workshop on Statistical Machine Translation